| Message ID | 20240404121521.22483-1-andrea.cervesato@suse.de |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [v2] New LTP documentation | expand |
Hi, after merging this patch, documentation website will be available at https://linux-test-project.readthedocs.io/en/latest/index.html Please update any patch you are working on and it's related to this one, because documentation has been moved from doc_new/ to doc/ folder. Andrea On 4/4/24 14:15, Andrea Cervesato wrote: > From: Andrea Cervesato <andrea.cervesato@suse.com> > > The new LTP documentation is meant to use Sphinx as the main > documentation framework, with ReconStructedText as the main language. > All the conversion has been done step-by-step, each chapter at time, > updating english syntax and fixing typos. There are minor improvements, > but overall the structure is the same. > The API chapters is ongoing and it will be completed soon in the future. > > The old documentation has been moved into doc/old folder, but it will > removed once new documentation will be completed. > > Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com> > --- > Moved documentation into doc/ folder > Moved old documentation into doc/old folder > > .readthedocs.yml | 18 + > README.md | 252 ---- > README.rst | 29 + > doc/.gitignore | 4 + > doc/Makefile | 15 +- > doc/_static/custom.css | 4 + > doc/conf.py | 201 ++++ > doc/developers/api_c_tests.rst | 7 + > doc/developers/api_kvm_tests.rst | 7 + > doc/developers/api_network_tests.rst | 7 + > doc/developers/api_shell_tests.rst | 4 + > doc/developers/build_system.rst | 213 ++++ > doc/developers/debugging.rst | 21 + > doc/developers/ltp_library.rst | 42 + > doc/developers/setup_mailinglist.rst | 50 + > doc/developers/test_case_tutorial.rst | 1038 +++++++++++++++++ > doc/developers/writing_tests.rst | 537 +++++++++ > doc/index.rst | 103 ++ > doc/maintainers/ltp_release_procedure.rst | 173 +++ > doc/maintainers/patch_review.rst | 171 +++ > doc/{ => old}/Build-System.asciidoc | 0 > doc/{ => old}/C-Test-API.asciidoc | 0 > doc/{ => old}/C-Test-Case-Tutorial.asciidoc | 0 > doc/{ => old}/C-Test-Network-API.asciidoc | 0 > doc/{ => old}/Contact-Info.asciidoc | 0 > doc/{ => old}/KVM-Test-API.asciidoc | 0 > ...TP-Library-API-Writing-Guidelines.asciidoc | 0 > doc/{ => old}/LTP-Release-Procedure.asciidoc | 0 > ...Maintainer-Patch-Review-Checklist.asciidoc | 0 > doc/old/Makefile | 11 + > doc/{ => old}/Shell-Test-API.asciidoc | 0 > ...-kernel,-libc,-toolchain-versions.asciidoc | 0 > .../Test-Writing-Guidelines.asciidoc | 0 > doc/{ => old}/User-Guidelines.asciidoc | 0 > doc/{ => old}/man1/Makefile | 0 > doc/{ => old}/man1/doio.1 | 0 > doc/{ => old}/man1/iogen.1 | 0 > doc/{ => old}/man1/ltp-bump.1 | 0 > doc/{ => old}/man1/ltp-pan.1 | 0 > doc/{ => old}/man3/Makefile | 0 > doc/{ => old}/man3/parse_opts.3 | 0 > doc/{ => old}/man3/parse_ranges.3 | 0 > doc/{ => old}/man3/random_range.3 | 0 > doc/{ => old}/man3/random_range_seed.3 | 0 > doc/{ => old}/man3/tst_res.3 | 0 > doc/{ => old}/man3/tst_sig.3 | 0 > doc/{ => old}/man3/tst_tmpdir.3 | 0 > doc/{ => old}/man3/usctest.3 | 0 > doc/{ => old}/namespaces-helper-tools.txt | 0 > doc/{ => old}/nommu-notes.txt | 0 > doc/{ => old}/rules.tsv | 0 > doc/requirements.txt | 3 + > doc/spelling_wordlist | 120 ++ > doc/users/quick_start.rst | 126 ++ > doc/users/setup_tests.rst | 104 ++ > doc/users/stats.rst | 9 + > doc/users/supported_systems.rst | 110 ++ > 57 files changed, 3118 insertions(+), 261 deletions(-) > create mode 100644 .readthedocs.yml > delete mode 100644 README.md > create mode 100644 README.rst > create mode 100644 doc/.gitignore > create mode 100644 doc/_static/custom.css > create mode 100644 doc/conf.py > create mode 100644 doc/developers/api_c_tests.rst > create mode 100644 doc/developers/api_kvm_tests.rst > create mode 100644 doc/developers/api_network_tests.rst > create mode 100644 doc/developers/api_shell_tests.rst > create mode 100644 doc/developers/build_system.rst > create mode 100644 doc/developers/debugging.rst > create mode 100644 doc/developers/ltp_library.rst > create mode 100644 doc/developers/setup_mailinglist.rst > create mode 100644 doc/developers/test_case_tutorial.rst > create mode 100644 doc/developers/writing_tests.rst > create mode 100644 doc/index.rst > create mode 100644 doc/maintainers/ltp_release_procedure.rst > create mode 100644 doc/maintainers/patch_review.rst > rename doc/{ => old}/Build-System.asciidoc (100%) > rename doc/{ => old}/C-Test-API.asciidoc (100%) > rename doc/{ => old}/C-Test-Case-Tutorial.asciidoc (100%) > rename doc/{ => old}/C-Test-Network-API.asciidoc (100%) > rename doc/{ => old}/Contact-Info.asciidoc (100%) > rename doc/{ => old}/KVM-Test-API.asciidoc (100%) > rename doc/{ => old}/LTP-Library-API-Writing-Guidelines.asciidoc (100%) > rename doc/{ => old}/LTP-Release-Procedure.asciidoc (100%) > rename doc/{ => old}/Maintainer-Patch-Review-Checklist.asciidoc (100%) > create mode 100644 doc/old/Makefile > rename doc/{ => old}/Shell-Test-API.asciidoc (100%) > rename doc/{ => old}/Supported-kernel,-libc,-toolchain-versions.asciidoc (100%) > rename doc/{ => old}/Test-Writing-Guidelines.asciidoc (100%) > rename doc/{ => old}/User-Guidelines.asciidoc (100%) > rename doc/{ => old}/man1/Makefile (100%) > rename doc/{ => old}/man1/doio.1 (100%) > rename doc/{ => old}/man1/iogen.1 (100%) > rename doc/{ => old}/man1/ltp-bump.1 (100%) > rename doc/{ => old}/man1/ltp-pan.1 (100%) > rename doc/{ => old}/man3/Makefile (100%) > rename doc/{ => old}/man3/parse_opts.3 (100%) > rename doc/{ => old}/man3/parse_ranges.3 (100%) > rename doc/{ => old}/man3/random_range.3 (100%) > rename doc/{ => old}/man3/random_range_seed.3 (100%) > rename doc/{ => old}/man3/tst_res.3 (100%) > rename doc/{ => old}/man3/tst_sig.3 (100%) > rename doc/{ => old}/man3/tst_tmpdir.3 (100%) > rename doc/{ => old}/man3/usctest.3 (100%) > rename doc/{ => old}/namespaces-helper-tools.txt (100%) > rename doc/{ => old}/nommu-notes.txt (100%) > rename doc/{ => old}/rules.tsv (100%) > create mode 100644 doc/requirements.txt > create mode 100644 doc/spelling_wordlist > create mode 100644 doc/users/quick_start.rst > create mode 100644 doc/users/setup_tests.rst > create mode 100644 doc/users/stats.rst > create mode 100644 doc/users/supported_systems.rst > > diff --git a/.readthedocs.yml b/.readthedocs.yml > new file mode 100644 > index 000000000..28e8c88a5 > --- /dev/null > +++ b/.readthedocs.yml > @@ -0,0 +1,18 @@ > +version: 2 > + > +build: > + os: "ubuntu-22.04" > + tools: > + python: "3.6" > + apt_packages: > + - enchant-2 > + - hunspell-en-us > + > +# Build from the doc/ directory with Sphinx > +sphinx: > + configuration: doc/conf.py > + > +# Explicitly set the version of Python and its requirements > +python: > + install: > + - requirements: doc/requirements.txt > diff --git a/README.md b/README.md > deleted file mode 100644 > index ef656eecd..000000000 > --- a/README.md > +++ /dev/null > @@ -1,252 +0,0 @@ > -Linux Test Project > -================== > - > -Linux Test Project is a joint project started by SGI, OSDL and Bull developed > -and maintained by IBM, Cisco, Fujitsu, SUSE, Red Hat, Oracle and others. The > -project goal is to deliver tests to the open source community that validate the > -reliability, robustness, and stability of Linux. > - > -The LTP testsuite contains a collection of tools for testing the Linux kernel > -and related features. Our goal is to improve the Linux kernel and system > -libraries by bringing test automation to the testing effort. Interested open > -source contributors are encouraged to join. > - > -Project pages are located at: http://linux-test-project.github.io/ > - > -The latest image is always available at: > -https://github.com/linux-test-project/ltp/releases > - > -The discussion about the project happens at LTP mailing list: > -http://lists.linux.it/listinfo/ltp > - > -LTP mailing list is archived at: > -https://lore.kernel.org/ltp/ > - > -IRC #ltp at: [irc.libera.chat](https://libera.chat/) > - > -The git repository is located at GitHub at: > -https://github.com/linux-test-project/ltp > - > -The patchwork instance is at: > -https://patchwork.ozlabs.org/project/ltp/list/ > - > -Warning! > -======== > - > -**Be careful with these tests!** > - > -Don't run them on production systems. Growfiles, doio, and iogen in particular > -stress the I/O capabilities of systems and while they should not cause problems > -on properly functioning systems, they are intended to find (or cause) problems. > - > -Quick guide to running the tests > -================================ > - > -If you have git, autoconf, automake, m4, pkgconf / pkg-config, libc headers, > -linux kernel headers and other common development packages installed (see > -INSTALL and ci/*.sh), the chances are the following will work: > - > -``` > -$ git clone https://github.com/linux-test-project/ltp.git > -$ cd ltp > -$ make autotools > -$ ./configure > -``` > - > -Now you can continue either with compiling and running a single test or with > -compiling and installing the whole testsuite. > - > -For optional library dependencies look into scripts for major distros in > -`ci/` directory. You can also build whole LTP with `./build.sh` script. > - > -Shortcut to running a single test > ---------------------------------- > -If you need to execute a single test you actually do not need to compile > -the whole LTP, if you want to run a syscall testcase following should work. > - > -``` > -$ cd testcases/kernel/syscalls/foo > -$ make > -$ PATH=$PATH:$PWD ./foo01 > -``` > - > -Shell testcases are a bit more complicated since these need a path to a shell > -library as well as to compiled binary helpers, but generally following should > -work. > - > -``` > -$ cd testcases/lib > -$ make > -$ cd ../commands/foo > -$ PATH=$PATH:$PWD:$PWD/../../lib/ ./foo01.sh > -``` > - > -Open Posix Testsuite has it's own build system which needs Makefiles to be > -generated first, then compilation should work in subdirectories as well. > - > -``` > -$ cd testcases/open_posix_testsuite/ > -$ make generate-makefiles > -$ cd conformance/interfaces/foo > -$ make > -$ ./foo_1-1.run-test > -``` > - > -Compiling and installing all testcases > --------------------------------------- > - > -``` > -$ make > -$ make install > -``` > - > -This will install LTP to `/opt/ltp`. > -* If you have a problem see `INSTALL` and `./configure --help`. > -* Failing that, ask for help on the mailing list or Github. > - > -Some tests will be disabled if the configure script can not find their build > -dependencies. > - > -* If a test returns `TCONF` due to a missing component, check the `./configure` > - output. > -* If a tests fails due to a missing user or group, see the Quick Start section > - of `INSTALL`. > - > -Running tests > -------------- > - > -To run all the test suites > - > -``` > -$ cd /opt/ltp > -$ ./runltp > -``` > - > -Note that many test cases have to be executed as root. > - > -To run a particular test suite > - > -``` > -$ ./runltp -f syscalls > -``` > - > -To run all tests with `madvise` in the name > - > -``` > -$ ./runltp -f syscalls -s madvise > -``` > -Also see > - > -``` > -$ ./runltp --help > -``` > - > -Test suites (e.g. syscalls) are defined in the runtest directory. Each file > -contains a list of test cases in a simple format, see doc/ltp-run-files.txt. > - > -Each test case has its own executable or script, these can be executed > -directly > - > -``` > -$ testcases/bin/abort01 > -``` > - > -Some have arguments > - > -``` > -$ testcases/bin/mesgq_nstest -m none > -``` > - > -The vast majority of test cases accept the -h (help) switch > - > -``` > -$ testcases/bin/ioctl01 -h > -``` > - > -Many require certain environment variables to be set > - > -``` > -$ LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh > -``` > - > -Most commonly, the path variable needs to be set and also `LTPROOT`, but there > -are a number of other variables, `runltp` usually sets these for you. > - > -Note that all shell scripts need the `PATH` to be set. However this is not > -limited to shell scripts, many C based tests need environment variables as > -well. > - > -For more info see `doc/User-Guidelines.asciidoc` or online at > -https://github.com/linux-test-project/ltp/wiki/User-Guidelines. > - > -Network tests > -------------- > -Network tests require certain setup, described in `testcases/network/README.md` > -(online at https://github.com/linux-test-project/ltp/tree/master/testcases/network). > - > -Containers > ----------- > - > -*Presently running the LTP inside a container is not a shortcut. It > -will make things much harder for you.* > - > -There is a Containerfile which can be used with Docker or > -Podman. Currently it can build Alpine and OpenSUSE images. > - > -The container can be built with a command like: > - > -``` > -$ podman build -t tumbleweed/ltp \ > - --build-arg PREFIX=registry.opensuse.org/opensuse/ \ > - --build-arg DISTRO_NAME=tumbleweed \ > - --build-arg DISTRO_RELEASE=20230925 . > -``` > - > -Or just `podman build .` which will create an Alpine container. > - > -It contains Kirk in /opt/kirk. So the following will run some tests. > - > -``` > -$ podman run -it --rm tumbleweed/ltp:latest > -$ cd /opt/kirk && ./kirk -f ltp -r syscalls > -``` > - > -SUSE also publishes a > -[smaller LTP container](https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3D%5Ebenchmark+container%3D.*) > -that is not based on the Containerfile. > - > -Debugging with gdb and strace > -============================= > - > -The new test library runs the actual test, i.e. the `test()` function in a > -forked process. To get stack trace of a crashing test in gdb it's needed to > -[`set follow-fork-mode child`](https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_25.html). > -To trace the test with strace, use strace with `-f` option to enable tracing of forked processes also. > - > -Developers corner > -================= > - > -Before you start you should read following documents: > - > -* `doc/Test-Writing-Guidelines.asciidoc` > -* `doc/Build-System.asciidoc` > -* `doc/LTP-Library-API-Writing-Guidelines.asciidoc` > - > -There is also a step-by-step tutorial: > - > -* `doc/C-Test-Case-Tutorial.asciidoc` > - > -If something is not covered there don't hesitate to ask on the LTP mailing > -list. Also note that these documents are available online at: > - > -* https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines > -* https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines > -* https://github.com/linux-test-project/ltp/wiki/Build-System > -* https://github.com/linux-test-project/ltp/wiki/C-Test-Case-Tutorial > - > -Although we accept GitHub pull requests, the preferred way is sending patches to our mailing list. > - > -It's a good idea to test patches on GitHub Actions before posting to mailing > -list. Our GitHub Actions setup covers various architectures and distributions in > -order to make sure LTP compiles cleanly on most common configurations. > -For testing you need to just push your changes to your own LTP fork on GitHub. > diff --git a/README.rst b/README.rst > new file mode 100644 > index 000000000..3e176bd0b > --- /dev/null > +++ b/README.rst > @@ -0,0 +1,29 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Linux Test Project > +================== > + > +Linux Test Project is a joint project started by SGI, OSDL and Bull developed > +and maintained by SUSE, Red Hat, Fujitsu, IBM, Cisco, Oracle and others. The > +project goal is to deliver tests to the open source community that validate > +reliability, robustness, and stability of the Linux Kernel. > + > +The testing suites contain a collection of tools for testing the Linux kernel > +and related features. Our goal is to improve the Linux kernel and system > +libraries by bringing test automation. > + > +.. warning:: > + > + LTP tests shouldn't run in production systems. In particular, > + growfiles, doio, and iogen, stress the I/O capabilities of the systems and > + they are intended to find (or cause) problems. > + > +Some references: > + > +* `Documentation <http://linux-test-project.rtfd.io/>`_ > +* `Source code <https://github.com/linux-test-project/ltp>`_ > +* `Releases <https://github.com/linux-test-project/ltp/releases>`_ > +* `Mailing List <http://lists.linux.it/listinfo/ltp>`_ > +* `Working patches (patchwork) <https://patchwork.ozlabs.org/project/ltp/list/>`_ > +* `Working patches (lore.kernel.org) <https://lore.kernel.org/ltp>`_ > +* `#ltp @ libera chat <https://libera.chat/>`_ > diff --git a/doc/.gitignore b/doc/.gitignore > new file mode 100644 > index 000000000..173179852 > --- /dev/null > +++ b/doc/.gitignore > @@ -0,0 +1,4 @@ > +html/ > +build/ > +_static/syscalls.rst > +syscalls.tbl > diff --git a/doc/Makefile b/doc/Makefile > index f7e4dd021..e536e95db 100644 > --- a/doc/Makefile > +++ b/doc/Makefile > @@ -1,11 +1,8 @@ > -# SPDX-License-Identifier: GPL-2.0-or-later > -# Copyright (C) 2009, Cisco Systems Inc. > -# Ngie Cooper, July 2009 > +all: > + sphinx-build -b html . html > > -top_srcdir ?= .. > +spelling: > + sphinx-build -b spelling -d build/doctree . build/spelling > > -include $(top_srcdir)/include/mk/env_pre.mk > - > -RECURSIVE_TARGETS := install > - > -include $(top_srcdir)/include/mk/generic_trunk_target.mk > +clean: > + rm -rf html/ > diff --git a/doc/_static/custom.css b/doc/_static/custom.css > new file mode 100644 > index 000000000..cb4d3821d > --- /dev/null > +++ b/doc/_static/custom.css > @@ -0,0 +1,4 @@ > +/* set multiline tables cells */ > +.wy-table-responsive table td { > + white-space: normal; > +} > diff --git a/doc/conf.py b/doc/conf.py > new file mode 100644 > index 000000000..fb3e83cf2 > --- /dev/null > +++ b/doc/conf.py > @@ -0,0 +1,201 @@ > +# Configuration file for the Sphinx documentation builder. > +# > +# For the full list of built-in configuration values, see the documentation: > +# https://www.sphinx-doc.org/en/master/usage/configuration.html > + > +import os > +import re > +import sphinx > +import socket > +import urllib.request > + > +# -- Project information ----------------------------------------------------- > +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information > + > +project = 'Linux Test Project' > +copyright = '2024, Linux Test Project' > +author = 'Linux Test Project' > +release = '1.0' > + > +# -- General configuration --------------------------------------------------- > +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration > + > +extensions = [ > + 'linuxdoc.rstKernelDoc', > + 'sphinxcontrib.spelling' > +] > + > +exclude_patterns = ["html*", '_static*'] > + > +spelling_lang = "en_US" > +spelling_warning = True > +spelling_exclude_patterns=['users/stats.rst'] > +spelling_word_list_filename = "spelling_wordlist" > + > +# -- Options for HTML output ------------------------------------------------- > +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output > + > +html_theme = 'sphinx_rtd_theme' > +html_static_path = ['_static'] > + > + > +def generate_syscalls_stats(_): > + """ > + Generate statistics for syscalls. We fetch the syscalls list from the kernel > + sources, then we compare it with testcases/kernel/syscalls folder and > + generate a file that is included in the statistics documentation section. > + """ > + output = '_static/syscalls.rst' > + > + # sometimes checking testcases/kernel/syscalls file names are not enough, > + # because in some cases (i.e. io_ring) syscalls are tested, but they are > + # part of a more complex scenario. In the following list, we define syscalls > + # which we know they are 100% tested already. > + white_list = [ > + 'rt_sigpending', > + 'sethostname', > + 'lsetxattr', > + 'inotify_add_watch', > + 'inotify_rm_watch', > + 'newfstatat', > + 'pselect6', > + 'fanotify_init', > + 'fanotify_mark', > + 'prlimit64', > + 'getdents64', > + 'pkey_mprotect', > + 'pkey_alloc', > + 'pkey_free', > + 'io_uring_setup', > + 'io_uring_enter', > + 'io_uring_register', > + 'epoll_pwait2', > + 'quotactl_fd', > + ] > + > + # populate with non-syscalls which are present in the kernel sources > + # syscalls file > + black_list = [ > + 'reserved177', > + 'reserved193', > + 'rseq' > + ] > + > + # fetch syscalls file > + error = False > + try: > + socket.setdefaulttimeout(3) > + urllib.request.urlretrieve( > + "https://raw.githubusercontent.com/torvalds/linux/master/arch/mips/kernel/syscalls/syscall_n64.tbl", > + "syscalls.tbl") > + except Exception as err: > + error = True > + logger = sphinx.util.logging.getLogger(__name__) > + msg = "Can't download syscall_n64.tbl from kernel sources" > + logger.warning(msg) > + > + with open(output, 'w+') as stats: > + stats.write(f".. warning::\n\n {msg}") > + > + if error: > + return > + > + text = [ > + 'Syscalls\n', > + '--------\n\n', > + ] > + > + # collect all available kernel syscalls > + regexp = re.compile(r'\d+\s+n64\s+(?P<syscall>\w+)\s+\w+') > + ker_syscalls = [] > + with open("syscalls.tbl", 'r') as data: > + for line in data: > + match = regexp.search(line) > + if match: > + ker_syscalls.append(match.group('syscall')) > + > + # collect all LTP tested syscalls > + ltp_syscalls = [] > + for root, _, files in os.walk('../testcases/kernel/syscalls'): > + for myfile in files: > + if myfile.endswith('.c'): > + ltp_syscalls.append(myfile) > + > + # compare kernel syscalls with LTP tested syscalls > + syscalls = {} > + for kersc in ker_syscalls: > + if kersc in black_list: > + continue > + > + if kersc not in syscalls: > + if kersc in white_list: > + syscalls[kersc] = True > + continue > + > + syscalls[kersc] = False > + > + for ltpsc in ltp_syscalls: > + if ltpsc.startswith(kersc): > + syscalls[kersc] = True > + > + # generate the statistics file > + tested_syscalls = [key for key, val in syscalls.items() if val] > + text.append('syscalls which are tested under **testcases/kernel/syscalls**:\n\n') > + text.append(f'* kernel syscalls: {len(ker_syscalls)}\n') > + text.append(f'* tested syscalls: {len(tested_syscalls)}\n\n') > + > + # create tested/untested syscalls table > + index_tested = 0 > + table_tested = [ > + 'Tested syscalls\n', > + '~~~~~~~~~~~~~~~\n\n', > + '.. list-table::\n', > + ' :header-rows: 0\n\n', > + ] > + > + index_untest = 0 > + table_untest = [ > + 'Untested syscalls\n', > + '~~~~~~~~~~~~~~~~~\n\n', > + '.. list-table::\n', > + ' :header-rows: 0\n\n', > + ] > + > + for sysname, tested in syscalls.items(): > + if tested: > + if (index_tested % 3) == 0: > + table_tested.append(f' * - {sysname}\n') > + else: > + table_tested.append(f' - {sysname}\n') > + > + index_tested += 1 > + else: > + if (index_untest % 3) == 0: > + table_untest.append(f' * - {sysname}\n') > + else: > + table_untest.append(f' - {sysname}\n') > + > + index_untest += 1 > + > + left = index_tested % 3 > + if left > 0: > + for index in range(0, left + 1): > + table_tested.append(f' -\n') > + > + left = index_untest % 3 > + if left > 0: > + for index in range(0, left + 1): > + table_untest.append(f' -\n') > + > + text.extend(table_tested) > + text.append('\n') > + text.extend(table_untest) > + > + # write the file > + with open(output, 'w+') as stats: > + stats.writelines(text) > + > + > +def setup(app): > + app.add_css_file('custom.css') > + app.connect('builder-inited', generate_syscalls_stats) > diff --git a/doc/developers/api_c_tests.rst b/doc/developers/api_c_tests.rst > new file mode 100644 > index 000000000..adee17f16 > --- /dev/null > +++ b/doc/developers/api_c_tests.rst > @@ -0,0 +1,7 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +.. Include headers in this file with: > +.. .. kernel-doc:: ../../include/tst_test.h > + > +Developing using C API > +====================== > diff --git a/doc/developers/api_kvm_tests.rst b/doc/developers/api_kvm_tests.rst > new file mode 100644 > index 000000000..5ec3fc3a4 > --- /dev/null > +++ b/doc/developers/api_kvm_tests.rst > @@ -0,0 +1,7 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +.. Include headers in this file with: > +.. .. kernel-doc:: ../../include/tst_test.h > + > +Developing using KVM API > +======================== > diff --git a/doc/developers/api_network_tests.rst b/doc/developers/api_network_tests.rst > new file mode 100644 > index 000000000..3e487d7f2 > --- /dev/null > +++ b/doc/developers/api_network_tests.rst > @@ -0,0 +1,7 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +.. Include headers in this file with: > +.. .. kernel-doc:: ../../include/tst_test.h > + > +Developing using network API > +============================ > diff --git a/doc/developers/api_shell_tests.rst b/doc/developers/api_shell_tests.rst > new file mode 100644 > index 000000000..2fb2a5068 > --- /dev/null > +++ b/doc/developers/api_shell_tests.rst > @@ -0,0 +1,4 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Developing using shell API > +========================== > diff --git a/doc/developers/build_system.rst b/doc/developers/build_system.rst > new file mode 100644 > index 000000000..1a6ad4c56 > --- /dev/null > +++ b/doc/developers/build_system.rst > @@ -0,0 +1,213 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Build system > +============ > + > +The following document briefly describes the steps and methodologies used for > +the new and improved Makefile system. > + > +The Problem > +----------- > + > +The problem with the old Makefile system is that it was very difficult to > +maintain and it lacked any sense of formal structure, thus developing for LTP > +and including new targets was more difficult than it should have been > +(maintenance). Furthermore, proper option-based cross-compilation was > +impossible due to the fact that the Makefiles didn't support a prefixing > +system, and the appropriate implicit / static rules hadn't been configured to > +compile into multiple object directories for out-of-tree build support (ease of > +use / functionality). Finally, there wasn't a means to setup dependencies > +between components, such that if a component required ``libltp.a`` in order to > +compile, it would go off and compile ``libltp.a`` first (ease of use). > + > +These items needed to be fixed to reduce maintenance nightmares for the > +development community contributing to LTP, and the project maintainers. > + > +Design > +------ > + > +The system was designed such that including a single GNU Makefile compatible > +set in each new directory component is all that's essentially required to > +build the system. > + > +Say you had a directory like the following (with ``.c`` files in them which > +directly tie into applications, e.g. baz.c -> baz): > + > +.. code-block:: > + > + .../foo/ > + |--> Makefile > + | > + --> bar/ > + | > + --> Makefile > + | > + --> baz.c > + > +.. code-block:: make > + :caption: .../foo/Makefile > + > + # > + # Copyright disclaimer goes here -- please use GPLv2. > + # > + > + top_srcdir ?= .. > + > + include $(top_srcdir)/include/mk/env_pre.mk > + include $(top_srcdir)/include/mk/generic_trunk_target.mk > + > +.. code-block:: make > + :caption: .../foo/bar/Makefile > + > + # > + # Copyright disclaimer goes here -- please use GPLv2. > + # > + > + top_srcdir ?= ../.. > + > + include $(top_srcdir)/include/mk/env_pre.mk > + include $(top_srcdir)/include/mk/generic_leaf_target.mk > + > +Kernel Modules > +-------------- > + > +Some of the tests need to build kernel modules, happily LTP has > +infrastructure for this. > + > +.. code-block:: make > + > + ifneq ($(KERNELRELEASE),) > + > + obj-m := module01.o > + > + else > + > + top_srcdir ?= ../../../.. > + include $(top_srcdir)/include/mk/testcases.mk > + > + REQ_VERSION_MAJOR := 2 > + REQ_VERSION_PATCH := 6 > + MAKE_TARGETS := test01 test02 module01.ko > + > + include $(top_srcdir)/include/mk/module.mk > + include $(top_srcdir)/include/mk/generic_leaf_target.mk > + > + endif > + > +This is a Makefile example that allows you to build kernel modules inside LTP. > +The prerequisites for the build are detected by the ``configure`` script. > + > +The ``REQ_VERSION_MAJOR`` and ``REQ_VERSION_PATCH`` describe minimal kernel > +version for which the build system tries to build the module. > + > +The build system is also forward compatible with changes in Linux kernel > +internal API so that, if module fails to build, the failure is ignored both on > +build and installation. If the userspace counterpart of the test fails to load > +the module because the file does not exists, the test is skipped. > + > +Note the ``ifneq($(KERNELRELEASE),)``. The reason it exists, it is that the > +Makefile is executed twice: once by LTP build system and once by kernel > +kbuild, see ``Documentation/kbuild/modules.rst`` in the Linux kernel tree for > +details on external module build. > + > +Make Rules and Make Variables > +----------------------------- > + > +When using make rules, avoid writing ad hoc rules like: > + > +.. code-block:: make > + > + [prog]: [dependencies] > + cc -I../../include $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(LDLIBS) \ > + -o [prog] [dependencies] > + > +This makes cross-compilation and determinism difficult, if not impossible. > +Besides, implicit rules are your friends and as long as you use ``MAKEOPTS=;`` > +in the top-level caller (or do ``$(subst r,$(MAKEOPTS)``) to remove ``-r``), > +the compile will complete successfully, assuming all other prerequisites have > +been fulfilled (libraries, headers, etc). > + > +.. list-table:: > + :header-rows: 1 > + > + * - Variable > + - Explaination > + > + * - $(AR) > + - The library archiver > + > + * - $(CC) > + - The system C compiler > + > + * - $(CCP) > + - The system C preprocessor > + > + * - $(CFLAGS) > + - C compiler flags > + > + * - $(CPPFLAGS) > + - Preprocessor flags, e.g. ``-I`` arguments > + > + * - $(DEBUG_CFLAGS) > + - Debug flags to pass to ``$(CC)``, ``-g``, etc > + > + * - $(KVM_LD) > + - Special linker for wrapping KVM payload binaries into linkable object > + files. Defaults to ``$(LD)``. Change this variable if the KVM Makefile > + fails to build files named ``*-payload.o`` > + > + * - $(LD) > + - The system linker (typically ``$(CC)``, but not necessarily) > + > + * - $(LDFLAGS) > + - What to pass in to the linker, including ``-L`` arguments and other ld > + arguments, apart from ``-l`` library includes (see ``$(LDLIBS)``). > + This should be done in the ``$(CC)`` args passing style when > + ``LD := $(CC)``, e.g. ``-Wl,-foo``, as opposed to ``-foo`` > + > + * - $(LDLIBS) > + - Libraries to pass to the linker (e.g. ``-lltp``, etc) > + > + * - $(LTPLDLIBS) > + - LTP internal libraries i.e. these in libs/ directory > + > + * - $(OPT_CFLAGS) > + - Optimization flags to pass into the C compiler, ``-O2``, etc. If you > + specify ``-O2`` or higher, you should also specify > + ``-fno-strict-aliasing``, because of gcc fstrict-aliasing optimization > + bugs in the tree optimizer. Search for **fstrict-aliasing optimization > + bug** with your favorite search engine. > + > + Examples of more recent bugs: tree-optimization/17510 > + and tree-optimization/39100. > + > + Various bugs have occurred in the past due to buggy logic in the > + tree-optimization portion of the gcc compiler, from 3.3.x to 4.4. > + > + * - $(RANLIB) > + - What to run after archiving a library > + > + * - $(WCFLAGS) > + - Warning flags to pass to ``$(CC)``, e.g. ``-Werror``, ``-Wall``, etc. > + > +Make System Variables > +--------------------- > + > +A series of variables are used within the make system that direct what actions > +need to be taken. Rather than listing the variables here, please refer to the > +comments contained in ``.../include/mk/env_pre.mk``. > + > +Guidelines and Recommendations > +------------------------------ > + > +Of course, GNU Make manual is the key to understand the Make system, but > +following manuals are probably the most important: > + > +* `Implicit Rules <http://www.gnu.org/software/make/manual/make.html#Implicit-Rules>`_ > +* `Variables and Expansion <http://www.gnu.org/software/make/manual/make.html#Using-Variables>`_ > +* `Origin Use <http://www.gnu.org/software/make/manual/make.html#Origin-Function>`_ > +* `VPath Use <http://www.gnu.org/software/make/manual/make.html#Directory-Search>`_ > + > +.. warning:: > + > + Rebuild from scratch before committing anything in the build system. > diff --git a/doc/developers/debugging.rst b/doc/developers/debugging.rst > new file mode 100644 > index 000000000..7a4e6b2ad > --- /dev/null > +++ b/doc/developers/debugging.rst > @@ -0,0 +1,21 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Debugging > +========= > + > +This section explains some tricks which can be used to debug test binaries. > + > +Debug messages > +-------------- > + > +The LTP framework currently supports ``TDEBUG`` flag test debug messages. These > +messages can be enabled using the ``-D`` test's argument. > + > +Tracing and debugging syscalls > +------------------------------ > + > +The new test library runs the actual test (i.e. the ``test()`` function) in a > +forked process. To get stack trace of a crashing test in gdb it's needed to > +`set follow-fork-mode child <https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_25.html>`_. > +To trace the test, please use ``strace -f`` to enable tracing also for the > +forked processes. > diff --git a/doc/developers/ltp_library.rst b/doc/developers/ltp_library.rst > new file mode 100644 > index 000000000..85b69c11c > --- /dev/null > +++ b/doc/developers/ltp_library.rst > @@ -0,0 +1,42 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +LTP Library guidelines > +====================== > + > +General Rules > +------------- > + > +When we extend library API, we need to apply the same general rules that we use > +when writing tests, plus: > + > +#. LTP library tests must go inside ``lib/newlib_tests`` directory > +#. LTP documentation has to be updated according to API changes > + > +Shell API > +--------- > + > +API source code is in ``tst_test.sh``, ``tst_security.sh`` and ``tst_net.sh`` > +(all in ``testcases/lib`` directory). > + > +Changes in the shell API should not introduce uncommon dependencies > +(use basic commands installed everywhere by default). > + > +Shell libraries > +~~~~~~~~~~~~~~~ > + > +Aside from shell API libraries in ``testcases/lib``, it's worth putting > +common code for a group of tests into a shell library. The filename > +should end with ``_lib.sh`` and the library should load ``tst_test.sh`` or > +``tst_net.sh``. > + > +Shell libraries should have conditional expansion for ``TST_SETUP`` or > +``TST_CLEANUP``, to avoid surprises when test specific setup/cleanup function is > +redefined by shell library. > + > +.. code-block:: bash > + > + # ipsec_lib.sh > + # SPDX-License-Identifier: GPL-2.0-or-later > + TST_SETUP="${TST_SETUP:-ipsec_lib_setup}" > + ... > + . tst_test.sh > diff --git a/doc/developers/setup_mailinglist.rst b/doc/developers/setup_mailinglist.rst > new file mode 100644 > index 000000000..1eee0ae9f > --- /dev/null > +++ b/doc/developers/setup_mailinglist.rst > @@ -0,0 +1,50 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Setting up the Mailing list > +=========================== > + > +Before using ``git send-email``, you need to set up your email client to send > +emails from the command line. This typically involves configuring an SMTP server > +and authentication details. > + > +Open a terminal and configure Git with your email settings using the following > +commands: > + > +.. code-block:: bash > + > + git config --global sendemail.from "Your Name <your_email@example.com>" > + git config --global sendemail.smtpserver "smtp.example.com" > + git config --global sendemail.smtpuser "your_email@example.com" > + git config --global sendemail.smtpserverport 587 > + git config --global sendemail.smtpencryption tls > + > +Replace ``smtp.example.com`` with the SMTP server address provided by your email > +provider. Replace ``your_email@example.com`` with your email address. Adjust the > +SMTP port and encryption settings according to your email provider's > +requirements. > + > +To test the configuration you can use ``--dry-run`` parameter. > + > +.. code-block:: bash > + > + git send-email --dry-run --to "ltp@lists.linux.it" --subject "Test Email" --body "This is a test email." HEAD^ > + > +Depending on your SMTP server's configuration, you may need to authenticate > +before sending emails. If required, configure authentication settings using: > + > +.. code-block:: bash > + > + git config --global sendemail.smtpuser "your_email@example.com" > + git config --global sendemail.smtppass "your_password" > + > +Replace ``your_email@example.com`` with your email address and ``your_password`` > +with your email account password. > + > +For any corner case, please take a look at the > +`email + git <https://git-send-email.io/>`_ documentation. > + > +.. note:: > + > + This method still works in most of the cases, but nowadays we often > + require to setup a two factor authentication. If this is the case, please > + consider setting up Git accordingly. > diff --git a/doc/developers/test_case_tutorial.rst b/doc/developers/test_case_tutorial.rst > new file mode 100644 > index 000000000..ae7d59959 > --- /dev/null > +++ b/doc/developers/test_case_tutorial.rst > @@ -0,0 +1,1038 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Test case tutorial > +================== > + > +This is a step-by-step tutorial on writing a simple C LTP test, where topics > +of the LTP and Linux kernel testing will be introduced gradually using a > +concrete example. Most sections will include exercises, some trivial and > +others not so much. If you find an exercise is leading you off at too much of > +a tangent, just leave it for later and move on. > + > +LTP tests can be written in C or Shell script. This tutorial is **only for tests > +written in C** using the new LTP test API. Note that while we go into some > +detail on using Git, this is not intended as a canonical or complete guide > +for Git. > + > +Assumptions & Feedback > +---------------------- > + > +We assume the reader is familiar with C, Git and common Unix/Linux/GNU tools > +and has some general knowledge of Operating Systems. Experienced Linux > +developers may find it too verbose while people new to system level Linux > +development may find it overwhelming. > + > +Comments and feedback are welcome, please direct them to the Mailing list. > + > +Getting Started > +--------------- > + > +First of all, make sure you have a copy of LTP in the current folder > +and we recommended cloning the Linux kernel repository for reference, this > +guide will refer to files and directories. > + > +.. code-block:: bash > + > + git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > + > +There are a number of other repositories which are useful for reference as > +well, including the GNU C library ``glibc`` and the alternative C library > +``musl``. Some system calls are partially or even entirely implemented in user > +land as part of the standard C library. So in these cases, the C library is an > +important reference. ``glibc`` is the most common C library for Linux, however > +``musl`` is generally easier to understand. > + > +How system calls are implemented varies from one architecture to another and > +across kernel and C library versions. To find out whether a system call is > +actually accessing the kernel (whether it is actually a system call) on any > +given machine you can use the ``strace`` utility. This intercepts system calls > +made by an executable and prints them. We will use this later in the tutorial. > + > +Choose a System Call to test > +---------------------------- > + > +We will use the ``statx()`` system call, to provide a concrete example of a > +test. At the time of writing there is no test for this call which was > +introduced in Linux kernel version 4.11. > + > +Linux system call specific tests are primarily contained in > +``testcases/kernel/syscalls``, but you should also ``git grep`` the entire LTP > +repository to check for any existing usages of a system call. > + > +One way to find a system call which is not currently tested by the LTP is to > +look at ``include/linux/syscalls.h`` in the kernel tree. > + > +Something the LTP excels to ensure bug-fixes are back ported to > +maintenance releases, so targeting a specific regression is another > +option. > + > +Find an untested System call > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Try to find an untested system call which has a manual page (i.e. ``man > +syscall`` produces a result). It is a good idea to Git-clone the latest kernel > +man-pages repository. > + > +.. code-block:: bash > + > + git clone git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git > + > +At the time of writing, the difference between the latest man-pages release and > +the ``HEAD`` of the repository (usually the latest commit) is well over 100 > +commits. This represents about 9 weeks of changes. If you are using a stable > +Linux distribution, your man-pages package may well be years old. So as with > +the kernel, it is best to have the Git repository as a reference. > + > +You could also find a system call with untested parameters or use whatever it > +is you are planning to use the LTP for. > + > +Create the test skeleton > +------------------------ > + > +I shall call my test ``statx01.c``, by the time you read this that file name > +will probably be taken, so increment the number in the file name as > +appropriate or replace ``statx`` with the system call in the chosen exercise. > + > +.. code-block:: bash > + > + mkdir testcases/kernel/syscalls/statx > + cd testcases/kernel/syscalls/statx > + echo statx >> .gitignore > + > +Next open ``statx01.c`` and add the following boilerplate. Make sure to change > +the copyright notice to your name/company, correct the test name and minimum > +kernel version if necessary. I will explain what the code does below. > + > +.. code-block:: c > + > + // SPDX-License-Identifier: GPL-2.0-or-later > + /* > + * Copyright (c) 2017 Instruction Ignorer <"can't"@be.bothered.com> > + */ > + > + /*\ > + * [Description] > + * > + * All tests should start with a description of _what_ we are testing. > + * Non-trivial explanations of _how_ the code works should also go here. > + * Include relevant links, Git commit hashes and CVE numbers. > + * Inline comments should be avoided. > + */ > + > + #include "tst_test.h" > + > + static void run(void) > + { > + tst_res(TPASS, "Doing hardly anything is easy"); > + } > + > + static struct tst_test test = { > + .test_all = run, > + .min_kver = "4.11", > + }; > + > +Starting with the ``#include`` statement we copy in the main LTP test library > +headers. This includes the most common test API functions and the test harness > +initialisation code. It is important to note that this is a completely > +ordinary, independent C program, however ``main()`` is missing because it is > +implemented in ``tst_test.h``. > + > +We specify what code we want to run as part of the test using the ``tst_test > +test`` structure. Various callbacks can be set by the test writer, including > +``test.test_all``, which we have set to ``run()``. The test harness will execute > +this callback in a separate process (using ``fork()``), forcibly terminating it > +if it does not return after ``test.timeout`` seconds. > + > +We have also set ``test.min_kver`` to the kernel version where ``statx`` was > +introduced. The test library will determine the kernel version at runtime. If > +the version is less than 4.11 then the test harness will return ``TCONF``, > +indicating that this test is not suitable for the current system > +configuration. > + > +Occasionally features are back ported to older kernel versions, so ``statx`` may > +exist on kernels with a lower version. However we don't need to worry about > +that unless there is evidence of it happening. > + > +As mentioned in the code itself, you should specify what you are testing and > +the expected outcome, even if it is relatively simple. If your program flow is > +necessarily complex and difficult to understand (which is often the case when > +trying to manipulate the kernel into doing something bad), then a detailed > +explanation of how the code works is welcome. > + > +What you should not do, is use inline comments or include the same level of > +explanation which is written here. As a general rule, if something is easy to > +document, then the code should also be easy to read. So don't document the easy > +stuff (except for the basic test specification). > + > +Before continuing we should compile this and check that the basics work. In > +order to compile the test we need a ``Makefile`` in the same subdirectory. If > +one already exists, then nothing needs to be done, otherwise add one with the > +following contents. > + > +.. code-block:: make > + > + # SPDX-License-Identifier: GPL-2.0-or-later > + # Copyright (c) 2019 Linux Test Project > + > + top_srcdir ?= ../../../.. > + > + include $(top_srcdir)/include/mk/testcases.mk > + > + include $(top_srcdir)/include/mk/generic_leaf_target.mk > + > +This will automatically add ``statx01.c`` as a build target producing a > +``statx01`` executable. Unless you have heavily deviated from the tutorial, and > +probably need to change ``top_srcdir``, nothing else needs to be done. > + > +Normally, if you were starting a Makefile from scratch, then you would need to > +add ``statx01`` as a build target. Specifying that you would like to run some > +program (e.g. ``gcc`` or ``clang``) to transform ``statx01.c`` into ``statx01``. > +Here we don't need to do that, but sometimes it is still necessary. For example, > +if we needed to link to the POSIX threading library, then we could add the > +following line after ``testcases.mk``. > + > +.. code-block:: make > + > + statx01: CFLAGS += -pthread > + > +Assuming you are in the test's subdirectory ``testcases/kernel/syscalls/statx``, > +please do: > + > +.. code-block:: bash > + > + make > + ./statx01 > + > +This should build the test and then run it. However, even though the test is > +in the ``syscalls`` directory it won't be automatically ran as part of the > +_syscalls_ test group (remember ``./runltp -f syscalls`` from the > +``README.md``?). For this we need to add it to the ``runtest`` file. So open > +``runtest/syscalls`` and add the lines starting with a ``+``. > + > +.. code-block:: > + > + statvfs01 statvfs01 > + statvfs02 statvfs02 > + > + +statx01 statx01 > + + > + stime01 stime01 > + stime02 stime02 > + > +The ``runtest`` files are in a two column format. The first column is the test > +name, which is mainly used by test runners for reporting and filtering. It is > +just a single string of text with no spaces. The second column, which can > +contain spaces, is passed to the shell in order to execute the test. Often it > +is just the executable name, but some tests also take arguments (the LTP has a > +library for argument parsing, by the way). > + > +If you haven't done so already, we should add all these new files to Git. It > +is vitally important that you do not make changes to the master branch. If you > +do then pulling changes from upstream becomes a major issue. So first of all > +create a new branch. > + > +.. code-block:: bash > + > + git checkout -b statx01 master > + > +Now we want to add the files we have created or modified, but before doing a > +commit make sure you have configured Git correctly. You need to at least set > +your Name and e-mail address in ``~/.gitconfig``, but there are some other > +settings which come in handy too. My relatively simple configuration is similar > +to the below: > + > +.. code-block:: ini > + > + [user] > + name = Sarah Jane > + email = sjane@e-mail.address > + [core] > + editor = emacs > + [sendemail] > + smtpServer = smtp.server.address > + > +Obviously you need to at least change your name and e-mail. The SMTP server is > +useful for ``git send-email``, which we will discuss later. The editor value is > +used for things like writing commits (without the ``-m`` option). > + > +.. code-block:: bash > + > + git add -v :/testcases/kernel/syscalls/statx :/runtest/syscalls > + git commit -m "statx01: Add new test for statx syscall" > + > +This should add all the new files in the ``statx`` directory and the ``runtest`` > +file. It is good practice to commit early and often. Later on we will do a > +Git-rebase, which allows us to clean up the commit history. So don't worry > +about how presentable your commit log is for now. Also don't hesitate to > +create a new branch when doing the exercises or experimenting. This will allow > +you to diverge from the tutorial and then easily come back again. > + > +I can't emphasize enough that Git makes things easy through branching and that > +things quickly get complicated if you don't do it. However if you do get into > +a mess, Git-reflog and Git-reset, will usually get you out of it. If you also > +mess that up then it may be possible to cherry pick 'dangling' commits out of > +the database into a branch. > + > +Report TCONF instead of TPASS > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Maybe the test should report ``TCONF: Not implemented`` instead or perhaps > +``TBROK``. Try changing it do so. > + > +Check Git ignores the executable > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Is your ``.gitignore`` correct? > + > +Run make check > +~~~~~~~~~~~~~~~~~~ > + > +Check coding style with ``make check``. > + > +Install the LTP and run the test with runtest > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Run ``statx01`` on its own, also using ``-I0`` amd ``-I10``. > + > +Call the system call > +-------------------- > + > +At the time of writing ``statx`` has no ``glibc`` wrapper. It is also fairly common > +for a distribution's C library version to be older than its kernel or it may use a > +cut down C library in comparison to the GNU one. So we must call ``statx()`` > +using the general ``syscall()`` interface. > + > +The LTP contains a library for dealing with the ``syscall`` interface, which is > +located in ``include/lapi``. System call numbers are listed against the relevant > +call in the ``*.in`` files (e.g. ``x86_64.in``) which are used to generate > +``syscalls.h``, which is the header you should include. On rare occasions you > +may find the system call number is missing from the ``*.in`` files and will need > +to add it (see ``include/lapi/syscalls/strip_syscall.awk``). > + > +System call numbers vary between architectures, hence there are multiple > +``*.in`` files for each architecture. You can find the various values for the > +``statx`` system call across a number of ``unistd.h`` files in the Linux kernel. > + > +Note that we don't use the system-call-identifier value available in > +``/usr/include/linux/uinstd.h`` because the kernel might be much newer than the > +user land development packages. > + > +For ``statx`` we had to add ``statx 332`` to ``include/lapi/syscalls/x86_64.in``, > +``statx 383`` to ``include/lapi/syscalls/powerpc.in``, etc. Now lets look at > +the code, which I will explain in more detail further down. > + > +.. code-block:: c > + > + /* > + * Test statx > + * > + * Check if statx exists and what error code it returns when we give it dodgy > + * data. > + */ > + > + #include <stdint.h> > + #include "tst_test.h" > + #include "lapi/syscalls.h" > + > + struct statx_timestamp { > + int64_t tv_sec; > + uint32_t tv_nsec; > + int32_t __reserved; > + }; > + > + struct statx { > + uint32_t stx_mask; > + uint32_t stx_blksize; > + uint64_t stx_attributes; > + uint32_t stx_nlink; > + uint32_t stx_uid; > + uint32_t stx_gid; > + uint16_t stx_mode; > + uint16_t __spare0[1]; > + uint64_t stx_ino; > + uint64_t stx_size; > + uint64_t stx_blocks; > + uint64_t stx_attributes_mask; > + struct statx_timestamp stx_atime; > + struct statx_timestamp stx_btime; > + struct statx_timestamp stx_ctime; > + struct statx_timestamp stx_mtime; > + uint32_t stx_rdev_major; > + uint32_t stx_rdev_minor; > + uint32_t stx_dev_major; > + uint32_t stx_dev_minor; > + uint64_t __spare2[14]; > + }; > + > + static int sys_statx(int dirfd, const char *pathname, int flags, > + unsigned int mask, struct statx *statxbuf) > + { > + return tst_syscall(__NR_statx, dirfd, pathname, flags, mask, statxbuf); > + } > + > + ... > + > +So the top part of the code is now boiler plate for calling ``statx``. It is > +common for the kernel to be newer than the user land libraries and headers. So > +for new system calls like ``statx``, we copy, with a few modifications, the > +relevant definitions into the LTP. This is somewhat like 'vendoring', although > +we are usually just copying headers required for interacting with the Kernel's > +ABI (Application Binary Interface), rather than internalising actual > +functionality. > + > +So from the top we include the ``stdint.h`` library which gives us the standard > +``(u)int*_t`` type definitions. We use these in place of the Kernel type > +definitions such as ``__u64`` in ``linux/types.h``. We then have a couple of > +structure definitions which form part of the ``statx`` API. These were copied > +from ``include/uapi/linux/stat.h`` in the Kernel tree. > + > +After that, there is a wrapper function, which saves us from writing > +``tst_syscall(__NR_statx, ...``, every time we want to make a call to > +``statx``. This also provides a stub for when ``statx`` is eventually integrated > +into the LTP library and also implemented by the C library. At that point we > +can switch to using the C library implementation if available or fallback to > +our own. > + > +The advantage of using the C library implementation is that it will often be > +better supported across multiple architectures. It will also mean we are using > +the system call in the same way most real programs would. Sometimes there are > +advantages to bypassing the C library, but in general it should not be our > +first choice. > + > +The final test should do a check during configuration (i.e. when we run > +``./configure`` before building) which checks if the ``statx`` system call and > +associated structures exists. This requires writing an ``m4`` file for use with > +``configure.ac`` which is processed during ``make autotools`` and produces the > +configure script. > + > +For the time being though we shall just ignore this. All you need to know for > +now is that this is a problem which eventually needs to be dealt with and that > +there is a system in place to handle it. > + > +.. code-block:: c > + > + ... > + > + static void run(void) > + { > + struct statx statxbuf = { 0 }; > + > + TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); > + > + if (TST_RET == 0) > + tst_res(TFAIL, "statx thinks it can stat NULL"); > + else if (TST_ERR == EFAULT) > + tst_res(TPASS, "statx set errno to EFAULT as expected"); > + else > + tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); > + } > + > + static struct tst_test test = { > + .test_all = run, > + .min_kver = "4.11", > + }; > + > +The ``TEST`` macro sets ``TST_RET`` to the return value of ``tst_statx()`` and > +``TST_ERR`` to the value of ``errno`` immediately after the functions > +return. This is mainly just for convenience, although it potentially could > +have other uses. > + > +We check whether the return value indicates success and if it doesn't also > +check the value of ``errno``. The last call to ``tst_res`` includes ``TERRNO``, > +which will print the current error number and associated description in > +addition to the message we have provided. Note that it uses the current value > +of ``errno`` not ``TST_ERR``. > + > +What we should have done in the example above is use ``TTERRNO`` which takes the > +value of ``TST_ERR``. > + > +If we try to run the test on a kernel where ``statx`` does not exist, then > +``tst_syscall`` will cause it to fail gracefully with ``TCONF``. Where ``TCONF`` > +indicates the test is not applicable to our configuration. > + > +The function ``tst_syscall`` calls ``tst_brk(TCONF,...)`` on failure. ``tst_brk`` > +causes the test to exit immediately, which prevents any further test code from > +being run. > + > +What are the differences between ``tst_brk`` and ``tst_res``? > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +See ``include/tst_test.h``. Also what do they have in common? > + > +What happens if you call ``tst_res(TINFO, ...)`` after ``sys_statx``? > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Does the test still function correctly? > + > +Extend the test to handle other basic error conditions > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +For example, see if you can trigger ``ENOENT`` instead. You shouldn't > +have to create any files, which is discussed in the next section. > + > +Setup, Cleanup and files > +------------------------ > + > +Some tests require resources to be allocated, or system settings to be > +changed, before the test begins. This ``setup`` only has to be done once at the > +beginning and at the end of the test needs to be removed or reverted. The > +``cleanup`` also has to be done regardless of whether the test breaks. > + > +Fortunately, like most test libraries, we have setup and cleanup (teardown) > +callbacks. ``setup`` is called once before ``run`` and ``cleanup`` is called once > +afterwards. Note that ``run`` itself can be called multiple times by the test > +harness, but that ``setup`` and ``cleanup`` are only called once. > + > +If either your code, a ``SAFE_*`` macro or a library function such as > +``tst_syscall`` call ``tst_brk``, then ``run`` will exit immediately and the > +``cleanup`` function is then called. Once ``cleanup`` is completed, the test > +executable will then exit altogether abandoning any remaining iterations of > +``run``. > + > +For ``statx`` we would like to create some files or file like objects which we > +have control over. Deciding where to create the files is easy, we just create > +it in the current working directory and let the LTP test harness handle where > +that should be by setting ``.needs_tmpdir = 1``. > + > +.. code-block:: c > + > + /* > + * Test statx > + * > + * Check if statx exists and what error code it returns when we give it dodgy > + * data. Then stat a file and check it returns success. > + */ > + > + #include <stdint.h> > + #include "tst_test.h" > + #include "lapi/syscalls.h" > + #include "lapi/fcntl.h" > + > + #define FNAME "file_to_stat" > + #define STATX_BASIC_STATS 0x000007ffU > + > + /*************** statx structure and wrapper goes here ! ***************/ > + ... > + > +We have added an extra include ``lapi/fcntl.h`` which wraps the system header by > +the same name (``#include <fcntl.h>``). This header ensures we have definitions > +for recently added macros such as ``AT_FDCWD`` by providing fall backs if the > +system header does not have them. The ``lapi`` directory contains a number of > +headers like this. > + > +At some point we may wish to add ``lapi/stat.h`` to provide a fall back for > +macros such as ``STATX_BASIC_STATS``. However for the time being we have just > +defined it in the test. > + > + > +.. code-block:: c > + > + ... > + > + static void setup(void) > + { > + SAFE_TOUCH(FNAME, 0777, NULL); > + } > + > + static void run(void) > + { > + struct statx statxbuf = { 0 }; > + > + TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); > + if (TST_RET == 0) > + tst_res(TFAIL, "statx thinks it can stat NULL"); > + else if (TST_ERR == EFAULT) > + tst_res(TPASS, "statx set errno to EFAULT as expected"); > + else > + tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); > + > + TEST(sys_statx(AT_FDCWD, FNAME, 0, STATX_BASIC_STATS, &statxbuf)); > + if (TST_RET == 0) > + tst_res(TPASS, "It returned zero so it must have worked!"); > + else > + tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); > + } > + > + static struct tst_test test = { > + .setup = setup, > + .test_all = run, > + .min_kver = "4.11", > + .needs_tmpdir = 1 > + }; > + > +The ``setup`` callback uses one of the LTP's ``SAFE`` functions to create an empty > +file ``file_to_stat``. Because we have set ``.needs_tmpdir``, we can just create > +this file in the present working directory. We don't need to create a > +``cleanup`` callback yet because the LTP test harness will recursively delete > +the temporary directory and its contents. > + > +The ``run`` function can be called multiple times by the test harness, however > +``setup`` and ``cleanup`` callbacks will only be ran once. > + > +.. warning:: > + > + By this point you may have begun to explore the LTP library headers or older > + tests. In which case you will have come across functions from the old API such > + as ``tst_brkm``. The old API is being phased out, so you should not use these > + functions. > + > +So far we haven't had to do any clean up. So our example doesn't answer the > +question "what happens if part of the clean up fails?". To answer this we are > +going to modify the test to ask the (highly contrived) question "What happens > +if I create and open a file, then create a hard-link to it, then call open > +again on the hard-link, then ``stat`` the file". > + > + > +.. code-block:: c > + > + #define LNAME "file_to_stat_link" > + > + ... > + > + static void setup(void) > + { > + fd = SAFE_OPEN(FNAME, O_CREAT, 0777); > + SAFE_LINK(FNAME, LNAME); > + lfd = SAFE_OPEN(LNAME, 0); > + } > + > + static void cleanup(void) > + { > + if (lfd != 0) > + SAFE_CLOSE(lfd); > + > + if (fd != 0) > + SAFE_CLOSE(fd); > + } > + > + static void run(void) > + { > + ... > + > + TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); > + if (TST_RET == 0) > + tst_res(TPASS, "It returned zero so it must have worked!"); > + else > + tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); > + } > + > + static struct tst_test test = { > + .setup = setup, > + .cleanup = cleanup, > + .test_all = run, > + .tcnt = 2, > + .min_kver = "4.11", > + .needs_tmpdir = 1 > + }; > + > +Because we are now opening a file, we need a ``cleanup`` function to close the > +file descriptors. We have to manually close the files to ensure the temporary > +directory is deleted by the test harness (see the > +https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[test > +writing guidelines] for details). > + > +As a matter of good practice, the file descriptors are closed in reverse > +order. In some circumstances the order in which clean up is performed is > +significant. In that case resources created towards the end of 'setup' are > +dependent on ones near the beginning. So during 'cleanup' we remove the > +dependants before their dependencies. > + > +If, for some reason, the file descriptor ``lfd`` became invalid during the test, > +but ``fd`` was still open, we do not want ``SAFE_CLOSE(lfd)`` to cause the > +``cleanup`` function to exit prematurely. If it did, then ``fd`` would remain open > +which would cause problems on some file systems. > + > +Nor do we want to call ``cleanup`` recursively. So during 'cleanup' ``tst_brk``, > +and consequently the ``SAFE`` functions, do not cause the test to exit with > +``TBROK``. Instead they just print an error message with ``TWARN``. > + > +It is not entirely necessary to check if the file descriptors have a none zero > +value before attempting to close them. However it avoids a bunch of spurious > +warning messages if we fail to open ``file_to_stat``. Test case failures can be > +difficult to interpret at the best of times, so avoid filling the log with > +noise. > + > +Check ``statx`` returns the correct number of hard links > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +The field ``statx.stx_nlink`` should be equal to 2, right? > + > +Git-branch > +~~~~~~~~~~ > + > +We are about to make some organisational changes to the test, so now would be > +a good time to branch. Then we can switch between the old and new versions, to > +check the behavior has not been changed by accident. > + > +Split the test > +-------------- > + > +In our current test, we have essentially rolled two different test cases into > +one. Firstly we check if an error is returned when bad arguments are provided > +and secondly we check what happens when we stat an actual file. Quite often it > +makes sense to call ``tst_res`` multiple times in a single test case because we > +are checking different properties of the same result, but here we are clearly > +testing two different scenarios. > + > +So we should split the test in two. One obvious way to do this is to create > +``statx02.c``, but that seems like overkill in order to separate two simple test > +cases. So, for now at least, we are going to do it a different way. > + > +.. code-block:: c > + > + ... > + > + static void run_stat_null(void) > + { > + struct statx statxbuf = { 0 }; > + > + TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); > + if (TST_RET == 0) > + tst_res(TFAIL, "statx thinks it can stat NULL"); > + else if (TST_ERR == EFAULT) > + tst_res(TPASS, "statx set errno to EFAULT as expected"); > + else > + tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); > + } > + > + static void run_stat_symlink(void) > + { > + struct statx statxbuf = { 0 }; > + > + TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); > + if (TST_RET == 0) > + tst_res(TPASS, "It returned zero so it must have worked!"); > + else > + tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); > + } > + > + static void run(unsigned int i) > + { > + switch(i) { > + case 0: run_stat_null(); > + case 1: run_stat_symlink(); > + } > + } > + > + static struct tst_test test = { > + .setup = setup, > + .cleanup = cleanup, > + .test = run, > + .tcnt = 2, > + .min_kver = "4.11", > + .needs_tmpdir = 1 > + }; > + > +So we have used an alternative form of the ``test`` or ``run`` callback which > +accepts an index. Some tests use this index with an array of parameters and > +expected return values. Others do something similar to the above. The index > +can be used how you want so long as each iteration calls ``tst_res`` in a > +meaningful way. > + > +If an iteration fails to return a result (i.e. call ``tst_res`` with a value > +other than ``TINFO``) then the test harness will report ``TBROK`` and print the > +iteration which failed. This prevents a scenario in your test from silently > +failing due to some faulty logic. > + > +What is wrong with the switch statement? > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Were you paying attention? Also see the output of ``make check``. > + > +Test a feature unique to statx > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +So far we have not tested anything which is unique to ``statx``. So, for > +example, you could check stx_btime is correct (possibly only to within a > +margin of error) and that it differs from ``stx_mtime`` after writing to the > +file. > + > +Alternatively you could check that ``stx_dev_major`` and ``stx_dev_minor`` are set > +correctly. Note that the LTP has helper functions for creating devices and > +file systems. > + > +This could be quite a challenging exercise. You may wish to tackle an > +altogether different test scenario instead. If you get stuck just move onto > +the next section and come back later. > + > +Submitting the test for review > +------------------------------ > + > +Ignoring the fact we should probably create ``lapi/stat.h`` along with a bunch > +of fallback logic in the build system. We can now get our test ready for > +submission. > + > +The first thing you need to do before considering submitting your test is run > +``make check-statx01`` or ``make check`` in the test's directory. Again, we use > +the kernel style guidelines where possible. Next you should create a new > +branch, this will allow you to reshape your commit history without fear. > + > +After that we have the pleasure of doing an interactive ``rebase`` to clean up > +our commit history. In its current form the test only really needs a single > +commit, but if you have been using Git correctly then you should have > +many. The main reason we want to compress it to a single commit, is to make > +the LTP's Git-log readable. It also allows us to write a coherent description > +of the work as a whole in retrospective. Although, when adding a new test, the > +test description in the code will probably make the commit message redundant. > + > +Anyway, as an example, we shall look at my personal commit history from this > +tutorial and ``rebase`` it. You should try following along with your own > +repository. First lets look at the commit history since we branched from > +master. > + > +.. code-block:: bash > + > + git log --oneline master..HEAD > + 152d39fe7 (HEAD -> tutorial-rebase2, tutorial-rebase) tutorial: Start Submitting patch section > + 70f7ce7ce statx01: Stop checkpatch from complaining > + bb0332bd7 tutorial: Fix review problems > + 6a87a084a statx01: Fix review problems > + d784b1e85 test-writing-guidelines: Remove old API argument > + c26e1be7a fixup! tutorial > + 1e24a5fb5 (me/tutorial-rebase) fixup! tutorial > + 568a3f7be fixup! tutorial > + 09dd2c829 statx: stage 6 > + bfeef7902 statx: stage 5b > + 76e03d714 statx: stage 5a > + 98f5bc7ac statx: stage 4 > + 6f8c16438 statx: stage 3 (Add statx01) > + 5d93b84d8 Add statx and other syscall numbers > + 5ca627b78 tutorial: Add a step-by-step C test tutorial > + > +So we have told git to show all the commits which don't exist in ``master``, but > +are in ``HEAD``, where ``HEAD`` is the top of the current branch. The current > +branch is ``tutorial-rebase2`` which I just created. I have already done one > +``rebase`` and submitted a patch for review, so my original branch was just called > +``tutorial``. > + > +As usual my commit history is starting to look like a bit of mess! There is > +even a commit in there which should not be in the this branch (Remove old API > +argument), however it can be ignored for now and 'cherry picked' into a new branch > +later. > + > +For my patch I actually need at least two commits, one which contains the > +tutorial text and one which contains the test and associated files. So first > +of all I want to 'squash' (amalgamate) all the commits appended with > +``tutorial:`` into the bottom commit. > + > +.. code-block:: bash > + > + $ git rebase -i 5ca627b78\^ > + ... > + > +This begins an interactive ``rebase`` where commit ``5ca6427b78`` is the earliest > +commit we want to edit. The ``^`` symbol after the commit hash, specifies the > +commit before this one. The interactive ``rebase`` command takes the last commit > +we want to keep unaltered as it's argument (in other words it takes a > +non-inclusive range). > + > +Upon entering a similar command you will be presented with a text file > +similar to the following. The file should be displayed in your text editor of > +choice, if it doesn't, then you may change the editor variable in > +``.gitconfig``. > + > +.. code-block:: bash > + > + pick 5ca627b78 tutorial: Add a step-by-step C test tutorial > + pick 5d93b84d8 Add statx and other syscall numbers > + pick 6f8c16438 statx: stage 3 (Add statx01) > + pick 98f5bc7ac statx: stage 4 > + pick 76e03d714 statx: stage 5a > + pick bfeef7902 statx: stage 5b > + pick 09dd2c829 statx: stage 6 > + pick 568a3f7be fixup! tutorial > + pick 1e24a5fb5 fixup! tutorial > + pick c26e1be7a fixup! tutorial > + pick d784b1e85 test-writing-guidelines: Remove old API argument > + pick 6a87a084a statx01: Fix review problems > + pick bb0332bd7 tutorial: Fix review problems > + pick 70f7ce7ce statx01: Stop checkpatch from complaining > + pick 152d39fe7 tutorial: Start Submitting patch section > + > +The last commit from Git-log is shown at the top. The left hand column > +contains the commands we want to run on each commit. ``pick`` just means we > +re-apply the commit as-is. We can reorder the lines to apply the commits in a > +different order, but we need to be careful when reordering commits to the same > +file. If your ``rebase`` results in a merge conflict, then you have probably > +reordered some commits which contained changes to the same piece of code. > + > +Perhaps a better name for the interactive ``rebase`` command would be 'replay'. As > +we pick a point in the commit history, undo all those commits before that > +point, then reapply them one at a time. During the replay we can reorder the > +commits, drop, merge, split and edit them, creating a new history. > + > +The commands I am going to use are ``reword`` and ``fixup``. The ``reword`` command > +allows you to edit a single commit's message. The 'fixup' command 'squashes' a > +commit into the commit above/preceding it, merging the two commits into > +one. The commit which has ``fixup`` applied has its commit message deleted. If > +you think a commit might have something useful in its message then you can use > +``squash`` instead. > + > +.. code-block:: bash > + > + reword 5ca627b78 tutorial: Add a step-by-step C test tutorial > + fixup 568a3f7be fixup! tutorial > + fixup 1e24a5fb5 fixup! tutorial > + fixup c26e1be7a fixup! tutorial > + fixup bb0332bd7 tutorial: Fix review problems > + fixup 152d39fe7 tutorial: Start Submitting patch section > + fixup 276edecab tutorial: Save changes before rebase > + pick 5d93b84d8 Add statx and other syscall numbers > + pick 6f8c16438 statx: stage 3 (Add statx01) > + pick 98f5bc7ac statx: stage 4 > + pick 76e03d714 statx: stage 5a > + pick bfeef7902 statx: stage 5b > + pick 09dd2c829 statx: stage 6 > + pick d784b1e85 test-writing-guidelines: Remove old API argument > + pick 6a87a084a statx01: Fix review problems > + > +So all the commits marked with ``fixup`` will be re-played by Git immediately > +after 5ca62 at the top. A new commit will then be created with the amalgamated > +changes of all the commits and 5ca62's log message. It turns out that I didn't > +need to reword anything, but there is no harm in checking. It is easy to > +forget the ``Signed-off-by:`` line. > + > +I could now do the same for the commits to the ``statx`` test, making the commit > +message prefixes consistent. However I am not actually going to submit the > +test (yet). > + > +I won't attempt to show you this, but if you need to do the opposite and split > +apart a commit. It is also possible using Git-rebase by marking a line with > +``edit``. This will pause Git just after replaying the marked commit. You can > +then use a 'soft' Git-reset to bring the selected commit's changes back into > +the 'index' where you are then able to un-stage some parts before > +re-committing. > + > +You can also use ``edit`` and ``git commit --amend`` together to change a commit > +deep in your history, but without resetting the 'index'. The 'index' contains > +changes which you have staged with ``git add``, but not yet committed. > + > +So now that the commit history has been cleaned up, we need to submit a patch > +to the mailing list or make a pull request on GitHub. The mailing list is the > +preferred place to make submissions and is more difficult for most people, so > +I will only cover that method. > + > +Just before we create the patch, we need to check that our changes will still > +apply to the master branch without problems. To do this we can use another > +type of ``rebase`` and then try rebuilding and running the test. > + > +.. code-block:: bash > + > + git checkout master > + git pull origin > + git checkout tutorial-rebase2 > + git rebase master > + > +Above, I update the master branch and then replay our changes onto it using > +``git rebase master``. You may find that after the rebase there is a merge > +conflict. This will result in something which looks like the following (taken > +from a Makefile conflict which was caused by reordering commits in a ``rebase``). > + > +.. code-block:: diff > + > + <<<<<<< HEAD > + cve-2016-7117: LDFLAGS += -lpthread > + ======= > + cve-2014-0196: LDFLAGS += -lpthread -lutil -lrt > + cve-2016-7117: LDFLAGS += -lpthread -lrt > + >>>>>>> 4dbfb8e79... Add -lrt > + > +The first line tells us this is the beginning of a conflict. The third line > +separates the two conflicting pieces of content and the last line is the end > +of the conflict. Usually, all you need to do is remove the lines you don't > +want, stage the changes and continue the ``rebase`` with ``git rebase > +--continue``. > + > +In order to create a patch e-mail we use > +`git format-patch <https://git-scm.com/docs/git-format-patch>`_, > +we can then send that e-mail using > +`git send-email <https://git-scm.com/docs/git-send-email>`_. > +It is also possible to import the patch (``mbox``) file into a number of e-mail > +programs. > + > +.. code-block:: bash > + > + $ git format-patch -1 -v 2 -o output --to ltp@lists.linux.it fd3cc8596 > + output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch > + > +The first argument ``-1`` specifies we want one commit from fd3cc8596 > +onwards. If we wanted this commit and the one after it we could specify ``-2`` > +instead. > + > +This is my second patch submission so I have used ``-v 2``, which indicates this > +is the second version of a patch set. The ``-o`` option specifies the output > +directory (literally called ``output``). The ``--to`` option adds the ``To:`` e-mail > +header, which I have set to the LTP mailing list. > + > +We can then send this patch with the following command sans ``--dry-run``. > + > +.. code-block:: bash > + > + git send-email --dry-run output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch > + > +Git will ask some questions (which you can ignore) and then tell you what it > +would do if this weren't a dry-run. In order for this to work you have to have > +a valid SMTP server set in ``.gitconfig`` and also be signed up to the LTP > +mailing list under the same e-mail address you have configured in Git. You can > +sign up at https://lists.linux.it/listinfo/ltp. > + > +Doing code review > +----------------- > + > +While waiting for your test to be reviewed, you are invited and encouraged to > +review other contributors' code. This may seem bizarre when you are completely > +new to the project, but there are two important ways in which you can > +contribute here: > + > +A. Point out logical errors in the code. > +B. Improve your own understanding > + > +It doesn't matter whether you know the canonical way of writing an LTP test in > +C. An error of logic, when properly explained, is usually indisputable. These > +are the most important errors to find as they always result in false test > +results. Once someone points out such an error it is usually obvious to > +everyone that it is a bug and needs to be fixed. > + > +Obviously testing the patch is one way of finding errors. You can apply > +patches using ``git am``. Then it is just a case of compiling and running the > +tests. > + > +Finally, reading and attempting to comment on other peoples patches, gives > +you a better understanding of the reviewers perspective. This is better for > +the project and for you. > + > +Style and organisational issues are best left to after you have found logical > +errors. > + > +Final notes > +----------- > + > +Hopefully you can now grasp the structure of an LTP test and have some idea of > +what is available in the LTP test library. There are a vast number of library > +functions available (mainly located in include and lib), some of which are > +documented in the test writing guidelines and many of which are not. > + > +We have only scratched the surface of the immense technical complexity of > +systems programming across multiple Kernel and C lib versions as well as > +different hardware architectures. The important thing to take away from this > +is that you have to be conscientious of what will happen on systems different > +from yours. The LTP has a huge and varied user base, so situations you may > +think are unlikely can and do happen to somebody. > + > +Of course you don't want to spend time allowing for situations which may never > +arise either, so you have to do your research and think about each situation > +critically. The more systems you can test on before submitting your changes, > +the better, although we understand not everyone has access to a lab. > + > +One important topic which has not been covered by this tutorial, is > +multi-process or multi-threaded testing. The LTP library functions work inside > +child processes and threads, but their semantics change slightly. There are > +also various helper functions for synchronising and forking processes. > + > +.. note:: > + > + When it comes time to submit a test, the preferred way to do it is on the > + mailing list although you can also use GitHub. The LTP follows similar rules > + to the kernel for formatting and submitting patches. Generally speaking the > + review cycle is easier for small patches, so try to make small changes or > + additions where possible. > diff --git a/doc/developers/writing_tests.rst b/doc/developers/writing_tests.rst > new file mode 100644 > index 000000000..7b4b70bd0 > --- /dev/null > +++ b/doc/developers/writing_tests.rst > @@ -0,0 +1,537 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Writing tests > +============= > + > +This document describes LTP guidelines and it's intended for anybody who wants > +to write or to modify a LTP testcase. It's not a definitive guide and it's not, > +by any means, a substitute for common sense. > + > +Guide to clean and understandable code > +-------------------------------------- > + > +Testcases require that the source code is easy to follow. When a test starts to > +fail, the failure has to be analyzed and clean test codebase makes this task > +much easier and quicker. > + > +Keep things simple > +~~~~~~~~~~~~~~~~~~ > + > +It's worth to keep testcases simple or, better, as simple as possible. > + > +The kernel and libc are tricky beasts and the complexity imposed by their > +interfaces is quite high. Concentrate on the interface you want to test and > +follow the UNIX philosophy. > + > +It's a good idea to make the test as self-contained as possible too, ideally > +tests should not depend on tools or libraries that are not widely available. > + > +Do not reinvent the wheel! > + > +* Use LTP standard interface > +* Do not add custom PASS/FAIL reporting functions > +* Do not write Makefiles from scratch, use LTP build system instead > +* Etc. > + > +Keep functions and variable names short > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Choosing a good name for an API functions or even variables is a difficult > +task do not underestimate it. > + > +There are a couple of customary names for different things that help people to > +understand code, for example: > + > +* For loop variables are usually named with a single letter ``i``, ``j``, ... > +* File descriptors ``fd`` or ``fd_foo``. > +* Number of bytes stored in file are usually named as ``size`` or ``len`` > +* Etc. > + > +Do not overcomment > +~~~~~~~~~~~~~~~~~~ > + > +Comments can sometimes save your day, but they can easily do more harm than > +good. There has been several cases where comments and actual implementation > +drifted slowly apart which yielded into API misuses and hard to find bugs. > +Remember there is only one thing worse than no documentation: wrong > +documentation. > + > +Ideally, everybody should write code that is obvious, which unfortunately isn't > +always possible. If there is a code that requires to be commented, keep it > +short and to the point. These comments should explain *why* and not *how* > +things are done. > + > +Never ever comment the obvious. > + > +In case of LTP testcases, it's customary to add an asciidoc formatted comment > +paragraph with highlevel test description at the beginning of the file right > +under the GPL SPDX header. This helps other people to understand the overall > +goal of the test before they dive into the technical details. It's also > +exported into generated documentation hence it should mostly explain what is > +tested. > + > +DRY (Code duplication) > +~~~~~~~~~~~~~~~~~~~~~~ > + > +Copy & paste is a good servant but very poor master. If you are about to copy a > +large part of the code from one testcase to another, think what would happen if > +you find bug in the code that has been copied all around the tree. What about > +moving it to a library instead? > + > +The same goes for short but complicated parts, whenever you are about to copy & > +paste a syscall wrapper that packs arguments accordingly to machine > +architecture or similarly complicated code, put it into a header instead. > + > +C coding style > +-------------- > + > +LTP adopted `Linux kernel coding style <https://www.kernel.org/doc/html/latest/process/coding-style.html>`_. > +Run ``make check`` in the test's directory and/or use ``make check-$TCID``, it > +uses (among other checks) our vendored version of > +`checkpatch.pl <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl>`_ > +script from kernel git tree. > + > +.. note:: > + If ``make check`` does not report any problems, the code still may be wrong > + as all tools used for checking only look for common mistakes. > + > +The following linting code can be found when we run ``make check``: > + > +.. list-table:: > + :header-rows: 1 > + > + * - Linting code > + - Message > + - Explanation > + > + * - LTP-001 > + - Library source files have ``tst_`` prefix > + - API source code is inside headers in ``include/{empty}*.h``, > + ``include/lapi/{empty}*.h`` (backward compatibility for old kernel and > + libc) and C sources in ``lib/{empty}*.c``. Files must have ``tst_`` > + prefix. > + > + * - LTP-002 > + - ``TST_RET`` and ``TST_ERR`` are never modified by test library functions > + - The test author is guaranteed that the test API will not modify these > + variables. This prevents silent errors where the return value and > + errno are overwritten before the test has chance to check them. > + > + The macros which are clearly intended to update these variables. That > + is ``TEST`` and those in ``tst_test_macros.h``. Are of course allowed to > + update these variables. > + > + * - LTP-003 > + - Externally visible library symbols have the ``tst_`` prefix > + - Functions, types and variables in the public test API should have the > + ``tst_`` prefix. With some exceptions for symbols already prefixed with > + ``safe_`` or ``ltp_``. > + > + Static (private) symbols should not have the prefix. > + > + * - LTP-004 > + - Test executable symbols are marked ``static`` > + - Test executables should not export symbols unnecessarily. This means > + that all top-level variables and functions should be marked with the > + ``static`` keyword. The only visible symbols should be those included > + from shared object files. > + > + * - LTP-005 > + - Array must terminate with a sentinel value (i.e. ``NULL`` or ``{}``) > + - When defining arrays in the ``struct tst_test`` structure, we need to > + end the array items with a sentinel ``NULL`` value. > + > +Shell coding style > +------------------ > + > +When writing testcases in shell, write in *portable shell* only, it's a good > +idea to try to run the test using alternative shell (alternative to bash, for > +example dash) too. > + > +*Portable shell* means Shell Command Language as defined by POSIX with an > +exception of few widely used extensions, namely **local** keyword used inside of > +functions and ``-o`` and ``-a`` test parameters (that are marked as obsolete in > +POSIX). > + > +You can either try to run the testcases in Debian which has ``/bin/sh`` pointing > +to ``dash`` by default, or to install ``dash`` on your favorite distribution, > +then use it to run tests. If your distribution lacks ``dash`` package, you can > +always compile it from `sources <http://gondor.apana.org.au/~herbert/dash/files/>`_. > + > +Run ``make check`` in the test's directory and/or use ``make check-$TCID.sh``. > +It uses (among other checks) our vendored version of > +`checkbashism.pl <https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl>`_ > +from Debian that is used to check for non-portable shell code. > + > +.. note:: > + > + If ``make check`` does not report any problems the code still may be wrong, > + as ``checkbashisms.pl`` is used for checking only common mistakes. > + > +Here there are some common sense style rules for shell > + > +* Keep lines under 80 chars > +* Use tabs for indentation > +* Keep things simple, avoid unnecessary subshells > +* Don't do confusing things (i.e. don't name your functions like common shell > + commands, etc.) > +* Quote variables > +* Be consistent > + > +3 Backwards compatibility > +~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +LTP test should be as backward compatible as possible. Think of an enterprise > +distributions with long term support (more than five years since the initial > +release) or of an embedded platform that needs to use several years old > +toolchain supplied by the manufacturer. > + > +Therefore LTP test for more current features should be able to cope with older > +systems. It should at least compile fine and if it's not appropriate for the > +configuration it should return ``TCONF``. > + > +There are several types of checks we use: > + > +* The *configure script* is usually used to detect availability of a function > + declarations in system headers. It's used to disable tests at compile time or > + to enable fallback definitions. > + > +* Checking the ``errno`` value is another type of runtime check. Most of the > + syscalls returns either ``EINVAL`` or ``ENOSYS`` when syscall was not > + implemented or was disabled upon kernel compilation. > + > +* LTP has kernel version detection that can be used to disable tests at runtime. > + Unfortunately, the kernel version does not always corresponds to a well > + defined feature set, as distributions tend to backport hundreds of patches > + while the kernel version stays the same. Use with caution. > + > +* Lately, we added a kernel ``.config`` parser. A test can define a boolean > + expression of kernel config variables that has to be satisfied in order to run > + a test. At the moment, this is mostly used for kernel namespaces. > + > +* Sometimes it makes sense to define a few macros instead of creating a > + configure test. One example is Linux specific POSIX clock ids in > + ``include/lapi/posix_clocks.h`` > + > +Dealing with messed up legacy code > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +LTP still contains a lot of old and messy code and we are cleaning it up as > +fast as we can but, despite the decade of efforts, there is still a lot of it. > +If you start modifying old or a messy testcase and your changes are more > +complicated than simple typo fixes, you should convert the test into a new > +library first. > + > +It's also much easier to review the changes if you split them into a smaller > +logical groups. The same goes for moving files: if you need to rename or to move > +files, do it in a separate patch. > + > +License > +~~~~~~~ > + > +Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or > +any later version). > + > +Use ``SPDX-License-Identifier: GPL-2.0-or-later`` > + > +LTP Structure > +------------- > + > +The structure of LTP is quite simple. Each test is a binary written either in > +portable shell or C. The test gets a configuration via environment variables > +and/or command line parameters, it prints additional information into the > +stdout and reports overall success/failure via the exit value. > + > +Tests are generally placed under the ``testcases/`` directory. Everything that > +is a syscall or (slightly confusingly) libc syscall wrapper, goes under > +``testcases/kernel/syscalls/``. > + > +There is also ``testcases/open_posix_testsuite/`` which is a well maintained > +fork of the Open POSIX testsuite project, that has been dead since 2005. > + > +We also have a number of directories with tests for more specific features, such > +as containers, etc. > + > +Runtest Files > +~~~~~~~~~~~~~ > + > +The list of tests to be executed is stored in runtest files under the > +``runtest/`` directory. The default set of runtest files to be executed is > +stored in ``scenario_groups/default``. When you add a test, you should add > +corresponding entries into some runtest file(s) as well. > + > +Each line of runtest file contains one test. The first item is the test name. > +All other items, separated by space will be executed as a command. > + > +.. code-block:: bash > + > + shell_test01 echo "SUCCESS" | shell_pipe01.sh > + splice02 splice02 -s 20 > + > +Blank lines and lines starting with a ``#`` (comments) are ignored. > + > +Syscalls tests, placed under ``testcases/kernel/syscalls/``, use > +``runtest/syscalls`` file. For kernel related tests for memory management we > +have ``runtest/mm``, etc. > + > +.. note:: > + > + runtest files should have one entry per a test. Creating a > + wrapper that runs all your tests and adding it as a single test > + into runtest file is strongly discouraged. > + > +Datafiles > +--------- > + > +If your test needs data files, these should be put into a subdirectory > +named ``datafiles`` and installed into the ``testcases/data/$TCID`` directory. > +This will require to add ``INSTALL_DIR := testcases/data/TCID`` into > +``datafiles/Makefile``. > + > +You can obtain path to datafiles via ``$TST_DATAROOT`` provided by ``test.sh`` > +or via C function ``tst_dataroot()`` provided by libltp: > + > +.. code-block:: c > + > + const char *dataroot = tst_dataroot(); > + > +Datafiles can also be accessed as ``$LTPROOT/testcases/data/$TCID/...``, > +but ``$TST_DATAROOT`` and ``tst_dataroot()`` are preferred, as these can be used > +when running testcases directly in git tree as well as from install location. > + > +Subexecutables > +~~~~~~~~~~~~~~ > + > +If your test needs to execute a binary, place it in the same directory of the > +testcase and name the binary with ``$TESTNAME_`` prefix, where ``$TESTNAME`` is > +the name of the test binary. Once the test is executed by the framework, the > +path to the directory with all LTP binaries is added to the ``$PATH`` and you > +can execute it via its name. > + > +.. note:: > + > + If you need to execute a test from the LTP tree, you can add ``PATH`` to > + the current directory with ``PATH="$PATH:$PWD" ./foo01``. > + > +Test Contribution Checklist > +--------------------------- > + > +#. Test compiles and it runs fine (check with ``-i 10`` and ``-i 0`` too) > +#. ``make check`` should not emit any warnings for the test you are working on > + (hint: run it in the test's directory and/or use ``make check-$TCID``) > +#. The runtest entries are in place > +#. New test binaries are added into the corresponding ``.gitignore`` files > +#. Patches apply over the latest git > + > +About .gitignore files > +~~~~~~~~~~~~~~~~~~~~~~ > + > +There are numerous ``.gitignore`` files in the LTP tree. Usually, there is a > +``.gitignore`` file for a group of tests. The reason of this setup is simple: > +it's easier to maintain a ``.gitignore`` file per tests' directory, rather > +than having a single file in the project root directory. In this way, we don't > +have to update all the gitignore files when moving directories, and they get > +deleted automatically when a directory with tests is removed. > + > +Testing pre-release kernel features > +----------------------------------- > + > +Tests for features not yet in the mainline kernel release are accepted. However, > +they must be added only to the **staging** runtest file. Once a feature is part > +of the stable kernel ABI, the associated test must be moved out of staging. > + > +Testing builds with GitHub Actions > +---------------------------------- > + > +Master branch is tested in `GitHub Actions <https://github.com/linux-test-project/ltp/actions>`_ > +to ensure LTP builds in various distributions, including old, current and > +bleeding edge. ``gcc`` and ``clang`` toolchains are also tested for various > +architectures using cross-compilation. For a full list of tested distros, please > +check ``.github/workflows/ci.yml``. > + > +.. note:: > + > + Passing the GitHub Actions CI means that LTP compiles in a variety of > + different distributions on their **newest releases**. > + The CI also checks for code linting, running ``make check`` in the whole > + LTP project. > + > +LTP C And Shell Test API Comparison > +----------------------------------- > + > +.. list-table:: > + :header-rows: 1 > + > + * - C API ``struct tst_test`` members > + - Shell API ``$TST_*`` variables > + > + * - .all_filesystems > + - TST_ALL_FILESYSTEMS > + > + * - .bufs > + - \- > + > + * - .caps > + - \- > + > + * - .child_needs_reinit > + - not applicable > + > + * - .cleanup > + - TST_CLEANUP > + > + * - .dev_extra_opts > + - TST_DEV_EXTRA_OPTS > + > + * - .dev_fs_opts > + - TST_DEV_FS_OPTS > + > + * - .dev_fs_type > + - TST_FS_TYPE > + > + * - .dev_min_size > + - not applicable > + > + * - .format_device > + - TST_FORMAT_DEVICE > + > + * - .max_runtime > + - \- > + > + * - .min_cpus > + - not applicable > + > + * - .min_kver > + - TST_MIN_KVER > + > + * - .min_mem_avail > + - not applicable > + > + * - .mnt_flags > + - TST_MNT_PARAMS > + > + * - .min_swap_avail > + - not applicable > + > + * - .mntpoint | .mnt_data > + - TST_MNTPOINT > + > + * - .mount_device > + - TST_MOUNT_DEVICE > + > + * - .needs_cgroup_ctrls > + - \- > + > + * - .needs_checkpoints > + - TST_NEEDS_CHECKPOINTS > + > + * - .needs_cmds > + - TST_NEEDS_CMDS > + > + * - .needs_devfs > + - \- > + > + * - .needs_device > + - TST_NEEDS_DEVICE > + > + * - .needs_drivers > + - TST_NEEDS_DRIVERS > + > + * - .needs_kconfigs > + - TST_NEEDS_KCONFIGS > + > + * - .needs_overlay > + - \- > + > + * - .needs_rofs > + - \- > + > + * - .needs_root > + - TST_NEEDS_ROOT > + > + * - .needs_tmpdir > + - TST_NEEDS_TMPDIR > + > + * - .options > + - TST_PARSE_ARGS | TST_OPTS > + > + * - .resource_files > + - \- > + > + * - .restore_wallclock > + - not applicable > + > + * - .sample > + - \- > + > + * - .save_restore > + - \- > + > + * - .scall > + - not applicable > + > + * - .setup > + - TST_SETUP > + > + * - .skip_filesystems > + - TST_SKIP_FILESYSTEMS > + > + * - .skip_in_compat > + - \- > + > + * - .skip_in_lockdown > + - TST_SKIP_IN_LOCKDOWN > + > + * - .skip_in_secureboot > + - TST_SKIP_IN_SECUREBOOT > + > + * - .supported_archs > + - not applicable > + > + * - .tags > + - \- > + > + * - .taint_check > + - \- > + > + * - .tcnt > + - TST_CNT > + > + * - .tconf_msg > + - not applicable > + > + * - .test | .test_all > + - TST_TESTFUNC > + > + * - .test_variants > + - \- > + > + * - .timeout > + - TST_TIMEOUT > + > + * - .tst_hugepage > + - not applicable > + > + * - .ulimit > + - not applicable > + > + * - not applicable > + - TST_NEEDS_KCONFIGS_IFS > + > + * - not applicable > + - TST_NEEDS_MODULE > + > + * - not applicable > + - TST_POS_ARGS > + > + * - not applicable > + - TST_USAGE > + > +.. list-table:: > + :header-rows: 1 > + > + * - C API other structs > + - Shell API ``$TST_*`` variables > + > + * - struct tst_device > + - TST_DEVICE > diff --git a/doc/index.rst b/doc/index.rst > new file mode 100644 > index 000000000..80820ab51 > --- /dev/null > +++ b/doc/index.rst > @@ -0,0 +1,103 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +.. include:: ../README.rst > + > +.. toctree:: > + :maxdepth: 3 > + :hidden: > + :caption: For users > + > + users/quick_start > + users/setup_tests > + users/supported_systems > + users/stats > + > +.. toctree:: > + :maxdepth: 3 > + :hidden: > + :caption: For developers > + > + developers/setup_mailinglist > + developers/writing_tests > + developers/test_case_tutorial > + developers/api_c_tests > + developers/api_shell_tests > + developers/api_network_tests > + developers/api_kvm_tests > + developers/ltp_library > + developers/build_system > + developers/debugging > + > +.. toctree:: > + :maxdepth: 3 > + :hidden: > + :caption: For maintainers > + > + maintainers/patch_review > + maintainers/ltp_release_procedure > + > +For users > +--------- > + > +.. descriptions here are active > + > +:doc:`users/quick_start` > + How to build and use LTP framework in few steps > + > +:doc:`users/setup_tests` > + How to setup tests execution > + > +:doc:`users/supported_systems` > + A list of supported technologies by the LTP framework > + > +:doc:`users/stats` > + Some LTP statistics > + > +For developers > +-------------- > + > +.. descriptions here are active > + > +:doc:`developers/setup_mailinglist` > + How to configure git and to start sending patches via ``git send-email`` > + > +:doc:`developers/writing_tests` > + Starting guide on writing tests > + > +:doc:`developers/test_case_tutorial` > + A tutorial showing how to write a test from scratch using C API > + > +:doc:`developers/api_c_tests` > + Walk through the C API features > + > +:doc:`developers/api_shell_tests` > + Walk through the Shell API features > + > +:doc:`developers/api_network_tests` > + Walk through the network API features > + > +:doc:`developers/api_kvm_tests` > + Walk through the KVM API features > + > +:doc:`developers/ltp_library` > + Developing new features in the LTP library > + > +:doc:`developers/build_system` > + Short introduction to the LTP build system > + > +:doc:`developers/debugging` > + How to debug LTP tests > + > +For maintainers > +--------------- > + > +:doc:`maintainers/patch_review` > + Steps to follow when reviewing patches > + > +:doc:`maintainers/ltp_release_procedure` > + Steps to follow for a new LTP release > + > + > +Getting help > +------------ > +To report a problem or suggest any feature, please write at ltp@lists.linux.it > diff --git a/doc/maintainers/ltp_release_procedure.rst b/doc/maintainers/ltp_release_procedure.rst > new file mode 100644 > index 000000000..f0bc01cee > --- /dev/null > +++ b/doc/maintainers/ltp_release_procedure.rst > @@ -0,0 +1,173 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Release process > +=============== > + > +Preparations > +------------ > + > +The release procedure generally takes a few weeks. In the first week or two, > +patches that should go into the release are reviewed and possibly merged. These > +patches are either fixes or patches pointed out by the community. > + > +Patch review, when finished, is followed by a git freeze, which is a period > +where only fixes are pushed to the git. During that period community is > +expected to run a LTP pre-release tests, reports problems, and/or send fixes to > +the mailing list. In this period we are especially making sure that there are > +no regressions in the test results on a wide range of distributions and > +architectures. > + > +Once the stabilization period has ended the time has finally come to proceed > +with the release. > + > +Prepare the release notes > +------------------------- > + > +Part of the preparation is also to write the release notes, which are then > +added to the GitHub release and also sent as announcement to various mailing > +lists (see below). > + > +Have a look at `this release letter <https://lore.kernel.org/ltp/ZGNiQ1sMGvPU_ETp@yuki/>`_ > +to get the idea how it should look. > + > +Tag the git and push changes to github > +-------------------------------------- > + > +.. code-block:: bash > + > + cd ltp > + echo YYYYMMDD > VERSION > + git commit -S -s -m 'LTP YYYYMMDD' VERSION > + git tag -s -a YYYYMMDD -m 'LTP YYYYMMDD' > + git push origin master:master > + git push origin YYYYMMDD > + > +The string ``YYYYMMDD`` should be substituted to the current date. > + > +You can use `tools/tag-release.sh <https://github.com/linux-test-project/ltp/blob/master/tools/tag-release.sh>`_ > +script to have the above automated process. > +It allows you to verify the tag before pushing it and does other checks. > + > +.. code-block:: bash > + > + $ ./tools/tag-release.sh > + ===== git push ===== > + new tag: 'YYYYMMDD', previous tag: '20230127' > + tag YYYYMMDD > + Tagger: Person-who-released LTP <foo@example.com> > + Date: ... > + > + LTP YYYYMMDD > + -----BEGIN PGP SIGNATURE----- > + ... > + -----END PGP SIGNATURE----- > + > + commit 3ebc2dfa85c2445bb68d8c0d66e33c4da1e1b3a7 > + gpg: using RSA key ... > + ... > + Primary key fingerprint: ... > + Author: Person-who-released LTP <foo@example.com> > + Date: ... > + > + LTP YYYYMMDD > + > + Signed-off-by: Person-who-released LTP <foo@example.com> > + > + diff --git a/VERSION b/VERSION > + index af4c41fec..ae488c0e7 100644 > + --- a/VERSION > + +++ b/VERSION > + @@ -1 +1 @@ > + -20230127 > + +YYYYMMDD > + > + Please check tag and signature. Proceed? [N/y]: y > + Pushing changes to upstream git. Proceed? [N/y]: y > + ... > + To github.com:linux-test-project/ltp.git > + * [new tag] YYYYMMDD -> YYYYMMDD > + > +Prepare tarballs and metadata documentation > +------------------------------------------- > + > +The following procedure will show how to create the release archives and the > +metadata documentation: > + > +.. code-block:: bash > + > + # clone already clonned git repository to new folder > + cd .. > + git clone ltp ltp-full-YYYYMMDD > + cd ltp-full-YYYYMMDD > + > + # update all submodules > + git submodule update --init > + > + # Generate configure script > + make autotools > + > + # Generate tarballs > + cd .. > + tar -cjf ltp-full-YYYYMMDD.tar.bz2 ltp-full-YYYYMMDD --exclude .git > + tar -cJf ltp-full-YYYYMMDD.tar.xz ltp-full-YYYYMMDD --exclude .git > + > + # Generate checksums > + md5 ltp-full-YYYYMMDD.tar.xz > ltp-full-YYYYMMDD.tar.xz.md5 > + sha1 ltp-full-YYYYMMDD.tar.xz > ltp-full-YYYYMMDD.tar.xz.sha1 > + sha256sum ltp-full-YYYYMMDD.tar.xz > ltp-full-YYYYMMDD.tar.xz.sha256 > + > + # Generate metadata documentation > + ./configure --with-metadata-generator=asciidoctor > + make -C metadata > + cp -v docparse/metadata.html ../metadata.YYYYMMDD.html > + > +You can use `tools/create-tarballs-metadata.sh <https://github.com/linux-test-project/ltp/blob/master/tools/create-tarballs-metadata.sh>`_ > +script to have the above procedure automated. All generated files are placed > +in the ``ltp-release-YYYYMMDD`` directory. > + > +.. code-block:: bash > + > + $ ./tools/create-tarballs-metadata.sh > + ===== git clone ===== > + Cloning into 'ltp-full-YYYYMMDD'... > + done. > + ===== Update submodules ===== > + Submodule 'tools/kirk' (https://github.com/linux-test-project/kirk.git) registered for path 'tools/kirk' > + ... > + ===== Generate configure script ===== > + sed -n '1{s:LTP-:m4_define([LTP_VERSION],[:;s:$:]):;p;q}' VERSION > m4/ltp-version.m4 > + aclocal -I m4 > + ... > + ===== Generate tarballs ===== > + ===== Generate checksums ===== > + ===== Generate metadata documentation ===== > + checking for a BSD-compatible install... /usr/bin/install -c > + ... > + 'docparse/metadata.html' -> '/home/foo/ltp-release-YYYYMMDD/metadata.YYYYMMDD.html' > + Generated files are in '/home/foo/ltp-release-YYYYMMDD', upload them to github > + > +Upload the generated files to GitHub > +------------------------------------ > + > +Click on `Releases <https://github.com/linux-test-project/ltp/releases>`_, then > +switch to `Tags <https://github.com/linux-test-project/ltp/tags>`_. Click on > +'Add release notes'. There should be 'Attach binaries ...' link at the > +bottom of the page. > + > +Don't forget to upload checksums for the tarballs and metadata documentation > +as well. > + > +Send release announcement > +------------------------- > + > +The announcement is sent to: > + > +* ltp at lists.linux.it > +* linux-kernel at vger.kernel.org > +* libc-alpha at sourceware.org > + > +CCed to: > + > +* lwn at lwn.net > +* akpm at linux-foundation.org > +* torvalds at linux-foundation.org > diff --git a/doc/maintainers/patch_review.rst b/doc/maintainers/patch_review.rst > new file mode 100644 > index 000000000..81d198532 > --- /dev/null > +++ b/doc/maintainers/patch_review.rst > @@ -0,0 +1,171 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Patch review > +============ > + > +Anyone can and should review patches. It's the only way to get good at patch > +review and for the project to scale. For this reason, we have a short guide on > +what to do during the review process. > + > +Goals of patch review > +--------------------- > + > +#. Prevent false positive test results > +#. Prevent false negative test results > +#. Keep the code as simple as possible, but no simpler > + > +How to find clear errors > +------------------------ > + > +A clear error is one where there is unlikely to be any argument if you > +provide evidence of it. Evidence being an error trace or logical proof > +that an error will occur in a common situation. > + > +The following are examples and may not be appropriate for all tests. > + > +* Merge the patch locally. It should apply cleanly to master. > +* Compile the patch with default and non-default configurations. > + > + * Use sanitizers e.g. undefined behaviour, address. > + * Compile on non-x86 > + * Compile on x86 with ``-m32`` > + * Compile testing patches with GitHub Actions in LTP repo fork can cover > + various distros/architectures > + > +* Use ``make check`` > +* Run effected tests in a VM > + > + * Use single vCPU > + * Use many vCPUs and enable NUMA > + * Restrict RAM to < 1GB. > + > +* Run effected tests on an embedded device > +* Run effected tests on non-x86 machine in general > +* Run reproducers on a kernel where the bug is present > +* Run tests with ``-i0`` and ``-i2`` > +* Compare usage of system calls with man page descriptions > +* Compare usage of system calls with kernel code > +* Double check commit message > +* Search the LTP library for existing helper functions > +* Check doc formatting, i.e. ``make doc && chromium docparse/metadata.html`` > + > +How to find subtle errors > +------------------------- > + > +A subtle error is one where you can expect some argument because you > +do not have clear evidence of an error. It is best to state these as > +questions and not make assertions if possible. > + > +Although if it is a matter of style or "taste" then senior maintainers > +can assert what is correct to avoid bike shedding. > + > +* Ask what happens if there is an error, could it be debugged just > + with the test output? > +* Are we testing undefined behavior? > + > + * Could future kernel behaviour change without "breaking userland"? > + * Does the kernel behave differently depending on hardware? > + * Does it behave differently depending on kernel configuration? > + * Does it behave differently depending on the compiler? > + * Would it behave differently if the order of checks on syscall parameters > + changed in the kernel? > + > +* Will it scale to tiny and huge systems? > + > + * What happens if there are 100+ CPUs? > + * What happens if each CPU core is very slow? > + * What happens if there are 2TB of RAM? > + > +* Are we repeating a pattern that can be turned into a library function? > +* Is a single test trying to do too much? > +* Could multiple similar tests be merged? > +* Race conditions > + > + * What happens if a process gets preempted? > + * Could checkpoints or fuzzsync by used instead? > + * Note, usually you can insert a sleep to prove a race condition > + exists however finding them is hard > + > +* Is there a simpler way to achieve the same kernel coverage? > + > +How to get patches merged > +------------------------- > + > +Once you think a patch is good enough you should add your ``Reviewed-by`` > +and/or ``Tested-by`` tags. This means you will get some credit for getting > +the patch merged. Also some blame if there are problems. > + > +If you ran the test you can add the ``Tested-by`` tag. If you read the > +code or used static analysis tools on it, you can add the Reviewed-by > +tag. > + > +In addition you can expect others to review your patches and add their > +tags. This will speed up the process of getting your patches merged. > + > +Maintainers Checklist > +--------------------- > + > +Patchset should be tested locally and ideally also in maintainer's fork in > +GitHub Actions on GitHub. > + > +.. note:: > + > + GitHub Actions do only build testing, passing the CI means only that > + the test compiles fine on variety of different distributions and releases. > + > +The test should be executed at least once locally and should PASS as well. > + > +Commit messages should have > + > +* Author's ``Signed-off-by`` tag > +* Committer's ``Reviewed-by`` or ``Signed-off-by`` tag > +* Check also mailing lists for other reviewers / testers tags, notes and failure > + reports > +* ``Fixes: hash`` if it fixes particular LTP commit > +* ``Fixes: #N`` if it fixes github issue number N, so it's automatically closed > +* LTP documentation should be kept up to date. > + > +After patch is accepted or rejected, set correct state and archive in the > +`LTP patchwork instance <https://patchwork.ozlabs.org/project/ltp/list/>`_. > + > +New tests > +--------- > + > +New test should > + > +* Have a record in runtest file > +* Test should work fine with more than one iteration (e.g. run with ``-i 100``) > +* Run with ``-i 0`` to check that setup and cleanup are coded properly > + (no test is being run) > +* Have a brief description > +* License: the default license for new tests is GPL v2 or later, use > + ``GPL-2.0-or-later``; the licence for test (e.g. GPL-2.0) should not change > + unless test is completely rewritten > +* Old copyrights should be kept unless test is completely rewritten > + > +C tests > +~~~~~~~ > + > +* Use the new C API, implementing ``struct tst_test`` > +* Test binaries are added into corresponding ``.gitignore`` files > +* Check coding style with ``make check`` > +* Docparse documentation > +* If a test is a regression test it should include ``.tags`` in the > + ``struct tst_test`` definition > +* When rewriting old tests, `uClinux <https://en.wikipedia.org/wiki/%CE%9CClinux>`_ > + support should be removed (project has been discontinued). > + E.g. remove ``#ifdef UCLINUX``, replace ``FORK_OR_VFORK()`` with simple > + ``fork()`` or ``SAFE_FORK()``. > + > +Shell tests > +~~~~~~~~~~~ > + > +* Use new shell API > +* Check coding style with ``make check`` > +* If a test is a regression test it should include related kernel or glibc > + commits as a comment > + > +LTP library > +~~~~~~~~~~~ > + > +For patchset touching the LTP library, follow the LTP library guidelines. > diff --git a/doc/Build-System.asciidoc b/doc/old/Build-System.asciidoc > similarity index 100% > rename from doc/Build-System.asciidoc > rename to doc/old/Build-System.asciidoc > diff --git a/doc/C-Test-API.asciidoc b/doc/old/C-Test-API.asciidoc > similarity index 100% > rename from doc/C-Test-API.asciidoc > rename to doc/old/C-Test-API.asciidoc > diff --git a/doc/C-Test-Case-Tutorial.asciidoc b/doc/old/C-Test-Case-Tutorial.asciidoc > similarity index 100% > rename from doc/C-Test-Case-Tutorial.asciidoc > rename to doc/old/C-Test-Case-Tutorial.asciidoc > diff --git a/doc/C-Test-Network-API.asciidoc b/doc/old/C-Test-Network-API.asciidoc > similarity index 100% > rename from doc/C-Test-Network-API.asciidoc > rename to doc/old/C-Test-Network-API.asciidoc > diff --git a/doc/Contact-Info.asciidoc b/doc/old/Contact-Info.asciidoc > similarity index 100% > rename from doc/Contact-Info.asciidoc > rename to doc/old/Contact-Info.asciidoc > diff --git a/doc/KVM-Test-API.asciidoc b/doc/old/KVM-Test-API.asciidoc > similarity index 100% > rename from doc/KVM-Test-API.asciidoc > rename to doc/old/KVM-Test-API.asciidoc > diff --git a/doc/LTP-Library-API-Writing-Guidelines.asciidoc b/doc/old/LTP-Library-API-Writing-Guidelines.asciidoc > similarity index 100% > rename from doc/LTP-Library-API-Writing-Guidelines.asciidoc > rename to doc/old/LTP-Library-API-Writing-Guidelines.asciidoc > diff --git a/doc/LTP-Release-Procedure.asciidoc b/doc/old/LTP-Release-Procedure.asciidoc > similarity index 100% > rename from doc/LTP-Release-Procedure.asciidoc > rename to doc/old/LTP-Release-Procedure.asciidoc > diff --git a/doc/Maintainer-Patch-Review-Checklist.asciidoc b/doc/old/Maintainer-Patch-Review-Checklist.asciidoc > similarity index 100% > rename from doc/Maintainer-Patch-Review-Checklist.asciidoc > rename to doc/old/Maintainer-Patch-Review-Checklist.asciidoc > diff --git a/doc/old/Makefile b/doc/old/Makefile > new file mode 100644 > index 000000000..f7e4dd021 > --- /dev/null > +++ b/doc/old/Makefile > @@ -0,0 +1,11 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# Copyright (C) 2009, Cisco Systems Inc. > +# Ngie Cooper, July 2009 > + > +top_srcdir ?= .. > + > +include $(top_srcdir)/include/mk/env_pre.mk > + > +RECURSIVE_TARGETS := install > + > +include $(top_srcdir)/include/mk/generic_trunk_target.mk > diff --git a/doc/Shell-Test-API.asciidoc b/doc/old/Shell-Test-API.asciidoc > similarity index 100% > rename from doc/Shell-Test-API.asciidoc > rename to doc/old/Shell-Test-API.asciidoc > diff --git a/doc/Supported-kernel,-libc,-toolchain-versions.asciidoc b/doc/old/Supported-kernel,-libc,-toolchain-versions.asciidoc > similarity index 100% > rename from doc/Supported-kernel,-libc,-toolchain-versions.asciidoc > rename to doc/old/Supported-kernel,-libc,-toolchain-versions.asciidoc > diff --git a/doc/Test-Writing-Guidelines.asciidoc b/doc/old/Test-Writing-Guidelines.asciidoc > similarity index 100% > rename from doc/Test-Writing-Guidelines.asciidoc > rename to doc/old/Test-Writing-Guidelines.asciidoc > diff --git a/doc/User-Guidelines.asciidoc b/doc/old/User-Guidelines.asciidoc > similarity index 100% > rename from doc/User-Guidelines.asciidoc > rename to doc/old/User-Guidelines.asciidoc > diff --git a/doc/man1/Makefile b/doc/old/man1/Makefile > similarity index 100% > rename from doc/man1/Makefile > rename to doc/old/man1/Makefile > diff --git a/doc/man1/doio.1 b/doc/old/man1/doio.1 > similarity index 100% > rename from doc/man1/doio.1 > rename to doc/old/man1/doio.1 > diff --git a/doc/man1/iogen.1 b/doc/old/man1/iogen.1 > similarity index 100% > rename from doc/man1/iogen.1 > rename to doc/old/man1/iogen.1 > diff --git a/doc/man1/ltp-bump.1 b/doc/old/man1/ltp-bump.1 > similarity index 100% > rename from doc/man1/ltp-bump.1 > rename to doc/old/man1/ltp-bump.1 > diff --git a/doc/man1/ltp-pan.1 b/doc/old/man1/ltp-pan.1 > similarity index 100% > rename from doc/man1/ltp-pan.1 > rename to doc/old/man1/ltp-pan.1 > diff --git a/doc/man3/Makefile b/doc/old/man3/Makefile > similarity index 100% > rename from doc/man3/Makefile > rename to doc/old/man3/Makefile > diff --git a/doc/man3/parse_opts.3 b/doc/old/man3/parse_opts.3 > similarity index 100% > rename from doc/man3/parse_opts.3 > rename to doc/old/man3/parse_opts.3 > diff --git a/doc/man3/parse_ranges.3 b/doc/old/man3/parse_ranges.3 > similarity index 100% > rename from doc/man3/parse_ranges.3 > rename to doc/old/man3/parse_ranges.3 > diff --git a/doc/man3/random_range.3 b/doc/old/man3/random_range.3 > similarity index 100% > rename from doc/man3/random_range.3 > rename to doc/old/man3/random_range.3 > diff --git a/doc/man3/random_range_seed.3 b/doc/old/man3/random_range_seed.3 > similarity index 100% > rename from doc/man3/random_range_seed.3 > rename to doc/old/man3/random_range_seed.3 > diff --git a/doc/man3/tst_res.3 b/doc/old/man3/tst_res.3 > similarity index 100% > rename from doc/man3/tst_res.3 > rename to doc/old/man3/tst_res.3 > diff --git a/doc/man3/tst_sig.3 b/doc/old/man3/tst_sig.3 > similarity index 100% > rename from doc/man3/tst_sig.3 > rename to doc/old/man3/tst_sig.3 > diff --git a/doc/man3/tst_tmpdir.3 b/doc/old/man3/tst_tmpdir.3 > similarity index 100% > rename from doc/man3/tst_tmpdir.3 > rename to doc/old/man3/tst_tmpdir.3 > diff --git a/doc/man3/usctest.3 b/doc/old/man3/usctest.3 > similarity index 100% > rename from doc/man3/usctest.3 > rename to doc/old/man3/usctest.3 > diff --git a/doc/namespaces-helper-tools.txt b/doc/old/namespaces-helper-tools.txt > similarity index 100% > rename from doc/namespaces-helper-tools.txt > rename to doc/old/namespaces-helper-tools.txt > diff --git a/doc/nommu-notes.txt b/doc/old/nommu-notes.txt > similarity index 100% > rename from doc/nommu-notes.txt > rename to doc/old/nommu-notes.txt > diff --git a/doc/rules.tsv b/doc/old/rules.tsv > similarity index 100% > rename from doc/rules.tsv > rename to doc/old/rules.tsv > diff --git a/doc/requirements.txt b/doc/requirements.txt > new file mode 100644 > index 000000000..742fb8b4b > --- /dev/null > +++ b/doc/requirements.txt > @@ -0,0 +1,3 @@ > +sphinx-rtd-theme==2.0.0 > +linuxdoc==20231020 > +sphinxcontrib-spelling==7.7.0 > diff --git a/doc/spelling_wordlist b/doc/spelling_wordlist > new file mode 100644 > index 000000000..0c016e615 > --- /dev/null > +++ b/doc/spelling_wordlist > @@ -0,0 +1,120 @@ > +aarch > +akpm > +amd > +archiver > +archs > +args > +asciidoc > +autoconf > +automake > +backport > +baz > +btime > +bufs > +CCed > +cgroup > +cgroups > +checksum > +checksums > +cmds > +codebase > +compat > +config > +cpus > +ctrls > +datafiles > +dev > +devfs > +dio > +distro > +distros > +docparse > +doio > +executables > +fd > +filesystem > +filesystems > +fixup > +fs > +fstrict > +fuzzsync > +gcc > +gdb > +github > +gitignore > +glibc > +growfiles > +hugepage > +iogen > +kbuild > +kconfigs > +kver > +ld > +le > +libc > +libltp > +libs > +linkable > +linux > +lockdown > +ltp > +lwn > +Maipo > +Makefile > +Makefiles > +mem > +mnt > +mntpoint > +msg > +namespace > +namespaces > +onwards > +openSUSE > +patchset > +pkgconf > +posix > +ppc > +pre > +pre > +preprocessor > +rebase > +reflog > +reinit > +repo > +rofs > +runtest > +scall > +secureboot > +sourceware > +statx > +stdout > +structs > +stx > +subdirectory > +subshell > +subshells > +syscall > +syscalls > +tcnt > +tconf > +teardown > +testcase > +testcases > +testsuite > +testsuites > +tmp > +tmpdir > +toolchain > +toolchains > +torvalds > +tst > +uClibc > +ulimit > +un > +userland > +userspace > +vCPU > +vCPUs > +vendoring > +vger > +wallclock > diff --git a/doc/users/quick_start.rst b/doc/users/quick_start.rst > new file mode 100644 > index 000000000..40d8dd9c6 > --- /dev/null > +++ b/doc/users/quick_start.rst > @@ -0,0 +1,126 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Installation and tests execution > +================================ > + > +Basics requirements to build LTP are the following: > + > +* git > +* autoconf > +* automake > +* make > +* gcc > +* m4 > +* pkgconf / pkg-config > +* libc headers > +* linux headers > + > +.. code-block:: console > + > + $ git clone https://github.com/linux-test-project/ltp.git > + $ cd ltp > + $ make autotools > + $ ./configure > + > +.. note:: > + > + For optional library dependencies, take a look at the scripts inside ``ci/`` > + directory. > + > +Running single tests > +-------------------- > + > +LTP provides the possibility to build and run single tests: > + > +.. code-block:: console > + > + $ cd testcases/kernel/syscalls/foo > + $ make > + $ PATH=$PATH:$PWD ./foo01 > + > +Shell testcases are a bit more complicated, since they need to setup ``PATH`` > +as well as to compiled binary helpers: > + > +.. code-block:: console > + > + $ cd testcases/lib > + $ make > + $ cd ../commands/foo > + $ PATH=$PATH:$PWD:$PWD/../../lib/ ./foo01.sh > + > +Open Posix Testsuite has it's own build system which needs Makefiles to be > +generated first: > + > +.. code-block:: console > + > + $ cd testcases/open_posix_testsuite/ > + $ make generate-makefiles > + $ cd conformance/interfaces/foo > + $ make > + $ ./foo_1-1.run-test > + > +Compiling and installing all testcases > +-------------------------------------- > + > +To compile all tests is really simple: > + > +.. code-block:: console > + > + $ make > + > + $ # install LTP inside /opt/ltp by default > + $ make install > + > +.. note:: > + > + Some tests will be disabled if ``configure`` script won't find the build > + dependencies. > + > +Running tests > +------------- > + > +To run all the test suites > + > +.. code-block:: console > + > + $ cd /opt/ltp > + > + $ # run syscalls testing suite > + $ ./kirk -f ltp -r syscalls > + > +.. note:: > + > + Many test cases have to be executed as root. > + > +Test suites (e.g. syscalls) are defined in the ``runtest`` directory. Each file > +contains a list of test cases in a simple format. > + > +Each test case has its own executable or script that can directly executed: > + > +.. code-block:: console > + > + $ testcases/bin/abort01 > + > + $ # some tests have arguments > + $ testcases/bin/mesgq_nstest -m none > + > + $ # vast majority of tests have a help > + $ testcases/bin/ioctl01 -h > + > + $ # Many require certain environment variables to be set > + $ LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh > + > +Most commonly, the ``PATH`` variable needs to be set and also ``LTPROOT``, but > +there are a number of other variables which usually ``kirk`` sets for you. > + > +.. note:: > + > + All shell scripts need the ``PATH`` to be set. However, this is not limited > + to shell scripts and some C based tests need environment variables as well. > + They usually raise a configuration error when this is needed. > + > +Network tests > +------------- > + > +Network tests usually require a certain setup that is described in > +``testcases/network/README.md``. > diff --git a/doc/users/setup_tests.rst b/doc/users/setup_tests.rst > new file mode 100644 > index 000000000..910cc2315 > --- /dev/null > +++ b/doc/users/setup_tests.rst > @@ -0,0 +1,104 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Tests setup > +=========== > + > +The internal LTP library provides a set of features that permits to customize > +tests behaviour by setting environment variables and using specific tests > +arguments. > + > +Library environment variables > +----------------------------- > + > +Following environment variables are expected to be set by LTP users. Therefore, > +with some exceptions, they have ``LTP_`` prefix. Environment variables with > +``TST_`` prefix are used inside LTP shell API and should **not** be set by > +users. > + > +.. list-table:: > + :header-rows: 1 > + > + * - Variable > + - Note > + > + * - KCONFIG_PATH > + - The path to the kernel config file, (if not set, it tries the usual paths > + ``/boot/config-RELEASE`` or ``/proc/config.gz``) > + > + * - KCONFIG_SKIP_CHECK > + - Skip kernel config check if variable set (not set by default) > + > + * - LTPROOT > + - Prefix for installed LTP. **Should be always set**, since some tests > + need it to use data files (``LTP_DATAROOT``). LTP is by default installed > + into ``/opt/ltp`` > + > + * - LTP_COLORIZE_OUTPUT > + - By default LTP colorizes it's output unless it's redirected to a pipe or > + file. Force colorized output behaviour: ``y`` or ``1``: always colorize, > + ``n`` or ``0``: never colorize. > + > + * - LTP_DEV > + - Path to the block device to be used. C Language: ``.needs_device = 1``. > + Shell language: ``TST_NEEDS_DEVICE=1``. > + > + * - LTP_SINGLE_FS_TYPE > + - Testing only - specifies filesystem instead all supported > + (for tests with ``.all_filesystems``). > + > + * - LTP_DEV_FS_TYPE > + - Filesystem used for testing (default: ``ext2``). > + > + * - LTP_TIMEOUT_MUL > + - Multiplies timeout, must be number >= 0.1 (> 1 is useful for slow > + machines to avoid unexpected timeout). > + > + * - LTP_RUNTIME_MUL > + - Multiplies maximal test iteration runtime. Tests that run for more than a > + second or two are capped on runtime. You can scale the default runtime > + both up and down with this multiplier. This is not yet implemented in the > + shell API. > + > + * - LTP_VIRT_OVERRIDE > + - Overrides virtual machine detection in the test library. Setting it to > + empty string, tella the library that system is not a virtual machine. > + Other possible values are ``kvm``, ``xen``, ``zvm`` and ``microsoft`` > + that describe different types supervisors. > + > + * - PATH > + - It's required to addjust path: ``PATH="$PATH:$LTPROOT/testcases/bin"`` > + > + * - TMPDIR > + - Base directory for template directory (C language: ``.needs_tmpdir = 1`` > + and shell: ``TST_NEEDS_TMPDIR=1``). Must be an absolute path (default: > + '/tmp'). > + > + * - LTP_NO_CLEANUP > + - Disable running test cleanup (defined in ``TST_CLEANUP``). > + Shell API only. > + > + * - LTP_ENABLE_DEBUG > + - Enable debug info (value ``1`` or ``y``). > + > +Test execution time and timeout > +------------------------------- > + > +The limit on how long a test can run does compose of two parts: ``max_runtime`` > +and ``timeout``. The limit does apply to a single test variant. That means, for > +example, that tests which run for all available filesystems will apply this > +limit for a single filesystem only. > + > +The ``max_runtime`` is a cap on how long the ``run()`` function can take and for > +most testcases this part is set to zero. For tests that do run for more than a > +second or two the ``max_runtime`` has to be defined and the ``run()`` function > +has to check actively how much runtime is left. > + > +Test runtime can be scaled up and down with ``LTP_RUNTIME_MUL`` environment > +variable or set on a commandline by the ``-I`` parameter. Hoewever, > +setting the runtime too low will cause long running tests to exit prematurely, > +possibly before having a chance to actually test anything. > + > +The timeout is a limit for test setup and cleanup and it's also a safety > +margin for the runtime accounting. It's currently set to 30 seconds but it may > +change later. If your target machine is too slow, it can be scaled up with the > +``LTP_TIMEOUT_MUL`` environment variable. > diff --git a/doc/users/stats.rst b/doc/users/stats.rst > new file mode 100644 > index 000000000..7073442aa > --- /dev/null > +++ b/doc/users/stats.rst > @@ -0,0 +1,9 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Statistics > +========== > + > +In this section we collect some statistics related to the current state of > +LTP tests. > + > +.. include:: ../_static/syscalls.rst > diff --git a/doc/users/supported_systems.rst b/doc/users/supported_systems.rst > new file mode 100644 > index 000000000..6220a96e0 > --- /dev/null > +++ b/doc/users/supported_systems.rst > @@ -0,0 +1,110 @@ > +.. SPDX-License-Identifier: GPL-2.0-or-later > + > +Supported systems > +================= > + > +Kernel version > +-------------- > + > +Minimal supported kernel version is **3.10**. > + > +Oldest tested distributions > +--------------------------- > + > +.. list-table:: > + :header-rows: 1 > + > + * - Distro > + - Kernel > + - glibc > + - gcc > + - clang > + > + * - openSUSE Leap 42.2 > + - 4.4 > + - 2.22 > + - 4.8.5 > + - \- > + > + * - Ubuntu 18.04 LTS bionic > + - 4.15 > + - 2.27 > + - 7.3.0 > + - \- > + > + * - Debian 11 (bullseye) > + - 5.10 > + - 2.31 > + - 10.2.1 > + - 11.0.1 > + > + * - RHEL-7.9 Maipo > + - 3.10 > + - 4.8.5 > + - 2.17 > + - \- > + > +Older distributions are not officially supported, which means that it > +may or may not work. It all depends on your luck. It should be possible > +to compile latest LTP even on slightly older distributions than we > +support with a few manual tweaks, e.g. disabling manually tests for > +newly added syscalls, etc. **Trivial fixes/workarounds may be accepted, > +but users are encouraged to move to a newer distro.** > + > +If latest LTP cannot be compiled even with some amount of workarounds, > +you may result to older LTP releases, however these are **not** supported > +in any way. Also if you are trying to run LTP on more than 10 years old > +distribution you may as well reconsider you life choices. > + > +Tested architectures > +-------------------- > + > +.. list-table:: > + :header-rows: 1 > + > + * - Architecture > + - Build > + > + * - x86_64 > + - native > + > + * - x86 emulation > + - native > + > + * - aarch64 > + - cross compilation > + > + * - ppc64le > + - cross compilation > + > + * - s390x > + - cross compilation > + > +Supported C libraries > +--------------------- > + > +.. list-table:: > + :header-rows: 1 > + > + * - C library > + - Note > + > + * - `glibc <https://www.gnu.org/software/libc/>`_ > + - Targeted libc, tested both compilation and actual test results. > + > + * - `uClibc-ng <https://uclibc-ng.org/>`_ > + - Although not being tested, it should work as it attempt to maintain a glibc compatible interface. > + > + * - `uClibc <https://www.uclibc.org/>`_ > + - Older uClibc might have problems. > + > + * - `musl <https://musl.libc.org/>`_ > + - Not yet fully supported. Check ``ci/alpine.sh`` script. > + > + * - Android > + - Please use `AOSP fork <https://android.googlesource.com/platform/external/ltp>`_ > + > +C version > +--------- > + > +LTP is compiled with ``-std=gnu99``.
Hi Andrea, > Hi, > after merging this patch, documentation website will be available at > https://linux-test-project.readthedocs.io/en/latest/index.html > Please update any patch you are working on and it's related to this one, > because documentation has been moved from doc_new/ to doc/ folder. I would put the old docs move into separate patch, but let's ignore it (less changes in a single commit). Acked-by: Petr Vorel <pvorel@suse.cz> Kind regards, Petr
Hi!
Can we please fix the spelling errors before this patchset is applied?
$ make spelling
...
WARNING: developers/build_system.rst:134: : Spell check: Explaination: Explaination.
WARNING: developers/test_case_tutorial.rst:136: : Spell check: initialisation: initialisation code. It is important to note that this is a completely.
WARNING: developers/test_case_tutorial.rst:378: : Spell check: internalising: ABI (Application Binary Interface), rather than internalising actual.
WARNING: developers/test_case_tutorial.rst:638: : Spell check: dependants: dependants before their dependencies..
WARNING: developers/test_case_tutorial.rst:663: : Spell check: organisational: We are about to make some organisational changes to the test, so now would be.
WARNING: developers/test_case_tutorial.rst:1004: : Spell check: organisational: Style and organisational issues are best left to after you have found logical.
WARNING: developers/test_case_tutorial.rst:1030: : Spell check: synchronising: also various helper functions for synchronising and forking processes..
WARNING: developers/writing_tests.rst:51: : Spell check: overcomment: Do not overcomment.
WARNING: developers/writing_tests.rst:67: : Spell check: highlevel: paragraph with highlevel test description at the beginning of the file right.
WARNING: developers/writing_tests.rst:89: : Spell check: vendored: uses (among other checks) our vendored version of.
WARNING: developers/writing_tests.rst:162: : Spell check: vendored: It uses (among other checks) our vendored version of.
WARNING: developers/writing_tests.rst:303: : Spell check: Subexecutables: Subexecutables.
WARNING: maintainers/patch_review.rst:29: : Spell check: behaviour: Use sanitizers e.g. undefined behaviour, address..
WARNING: maintainers/patch_review.rst:66: : Spell check: behaviour: Could future kernel behaviour change without "breaking userland"?.
WARNING: maintainers/patch_review.rst:141: : Spell check: licence: ; the licence for test (e.g. GPL-2.0) should not change.
WARNING: users/setup_tests.rst:7: : Spell check: behaviour: tests behaviour by setting environment variables and using specific tests.
WARNING: users/setup_tests.rst:38: : Spell check: behaviour: file. Force colorized output behaviour: .
WARNING: users/setup_tests.rst:64: : Spell check: tella: empty string, tella the library that system is not a virtual machine..
WARNING: users/setup_tests.rst:69: : Spell check: addjust: It's required to addjust path: .
WARNING: users/setup_tests.rst:97: : Spell check: commandline: variable or set on a commandline by the .
WARNING: users/setup_tests.rst:96: : Spell check: Hoewever: parameter. Hoewever,.
WARNING: Found 21 misspelled words
build succeeded, 22 warnings.
Most of these are actually valid typos.
Other than that:
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Hi, On 4/4/24 15:04, Cyril Hrubis wrote: > Hi! > Can we please fix the spelling errors before this patchset is applied? > > $ make spelling > ... > WARNING: developers/build_system.rst:134: : Spell check: Explaination: Explaination. > WARNING: developers/test_case_tutorial.rst:136: : Spell check: initialisation: initialisation code. It is important to note that this is a completely. > WARNING: developers/test_case_tutorial.rst:378: : Spell check: internalising: ABI (Application Binary Interface), rather than internalising actual. > WARNING: developers/test_case_tutorial.rst:638: : Spell check: dependants: dependants before their dependencies.. > WARNING: developers/test_case_tutorial.rst:663: : Spell check: organisational: We are about to make some organisational changes to the test, so now would be. > WARNING: developers/test_case_tutorial.rst:1004: : Spell check: organisational: Style and organisational issues are best left to after you have found logical. > WARNING: developers/test_case_tutorial.rst:1030: : Spell check: synchronising: also various helper functions for synchronising and forking processes.. > WARNING: developers/writing_tests.rst:51: : Spell check: overcomment: Do not overcomment. > WARNING: developers/writing_tests.rst:67: : Spell check: highlevel: paragraph with highlevel test description at the beginning of the file right. > WARNING: developers/writing_tests.rst:89: : Spell check: vendored: uses (among other checks) our vendored version of. > WARNING: developers/writing_tests.rst:162: : Spell check: vendored: It uses (among other checks) our vendored version of. > WARNING: developers/writing_tests.rst:303: : Spell check: Subexecutables: Subexecutables. > WARNING: maintainers/patch_review.rst:29: : Spell check: behaviour: Use sanitizers e.g. undefined behaviour, address.. > WARNING: maintainers/patch_review.rst:66: : Spell check: behaviour: Could future kernel behaviour change without "breaking userland"?. > WARNING: maintainers/patch_review.rst:141: : Spell check: licence: ; the licence for test (e.g. GPL-2.0) should not change. > WARNING: users/setup_tests.rst:7: : Spell check: behaviour: tests behaviour by setting environment variables and using specific tests. > WARNING: users/setup_tests.rst:38: : Spell check: behaviour: file. Force colorized output behaviour: . > WARNING: users/setup_tests.rst:64: : Spell check: tella: empty string, tella the library that system is not a virtual machine.. > WARNING: users/setup_tests.rst:69: : Spell check: addjust: It's required to addjust path: . > WARNING: users/setup_tests.rst:97: : Spell check: commandline: variable or set on a commandline by the . > WARNING: users/setup_tests.rst:96: : Spell check: Hoewever: parameter. Hoewever,. > WARNING: Found 21 misspelled words > build succeeded, 22 warnings. > > Most of these are actually valid typos. > > Other than that: > > Reviewed-by: Cyril Hrubis <chrubis@suse.cz> > Sure, good idea. Andrea
diff --git a/.readthedocs.yml b/.readthedocs.yml new file mode 100644 index 000000000..28e8c88a5 --- /dev/null +++ b/.readthedocs.yml @@ -0,0 +1,18 @@ +version: 2 + +build: + os: "ubuntu-22.04" + tools: + python: "3.6" + apt_packages: + - enchant-2 + - hunspell-en-us + +# Build from the doc/ directory with Sphinx +sphinx: + configuration: doc/conf.py + +# Explicitly set the version of Python and its requirements +python: + install: + - requirements: doc/requirements.txt diff --git a/README.md b/README.md deleted file mode 100644 index ef656eecd..000000000 --- a/README.md +++ /dev/null @@ -1,252 +0,0 @@ -Linux Test Project -================== - -Linux Test Project is a joint project started by SGI, OSDL and Bull developed -and maintained by IBM, Cisco, Fujitsu, SUSE, Red Hat, Oracle and others. The -project goal is to deliver tests to the open source community that validate the -reliability, robustness, and stability of Linux. - -The LTP testsuite contains a collection of tools for testing the Linux kernel -and related features. Our goal is to improve the Linux kernel and system -libraries by bringing test automation to the testing effort. Interested open -source contributors are encouraged to join. - -Project pages are located at: http://linux-test-project.github.io/ - -The latest image is always available at: -https://github.com/linux-test-project/ltp/releases - -The discussion about the project happens at LTP mailing list: -http://lists.linux.it/listinfo/ltp - -LTP mailing list is archived at: -https://lore.kernel.org/ltp/ - -IRC #ltp at: [irc.libera.chat](https://libera.chat/) - -The git repository is located at GitHub at: -https://github.com/linux-test-project/ltp - -The patchwork instance is at: -https://patchwork.ozlabs.org/project/ltp/list/ - -Warning! -======== - -**Be careful with these tests!** - -Don't run them on production systems. Growfiles, doio, and iogen in particular -stress the I/O capabilities of systems and while they should not cause problems -on properly functioning systems, they are intended to find (or cause) problems. - -Quick guide to running the tests -================================ - -If you have git, autoconf, automake, m4, pkgconf / pkg-config, libc headers, -linux kernel headers and other common development packages installed (see -INSTALL and ci/*.sh), the chances are the following will work: - -``` -$ git clone https://github.com/linux-test-project/ltp.git -$ cd ltp -$ make autotools -$ ./configure -``` - -Now you can continue either with compiling and running a single test or with -compiling and installing the whole testsuite. - -For optional library dependencies look into scripts for major distros in -`ci/` directory. You can also build whole LTP with `./build.sh` script. - -Shortcut to running a single test ---------------------------------- -If you need to execute a single test you actually do not need to compile -the whole LTP, if you want to run a syscall testcase following should work. - -``` -$ cd testcases/kernel/syscalls/foo -$ make -$ PATH=$PATH:$PWD ./foo01 -``` - -Shell testcases are a bit more complicated since these need a path to a shell -library as well as to compiled binary helpers, but generally following should -work. - -``` -$ cd testcases/lib -$ make -$ cd ../commands/foo -$ PATH=$PATH:$PWD:$PWD/../../lib/ ./foo01.sh -``` - -Open Posix Testsuite has it's own build system which needs Makefiles to be -generated first, then compilation should work in subdirectories as well. - -``` -$ cd testcases/open_posix_testsuite/ -$ make generate-makefiles -$ cd conformance/interfaces/foo -$ make -$ ./foo_1-1.run-test -``` - -Compiling and installing all testcases --------------------------------------- - -``` -$ make -$ make install -``` - -This will install LTP to `/opt/ltp`. -* If you have a problem see `INSTALL` and `./configure --help`. -* Failing that, ask for help on the mailing list or Github. - -Some tests will be disabled if the configure script can not find their build -dependencies. - -* If a test returns `TCONF` due to a missing component, check the `./configure` - output. -* If a tests fails due to a missing user or group, see the Quick Start section - of `INSTALL`. - -Running tests -------------- - -To run all the test suites - -``` -$ cd /opt/ltp -$ ./runltp -``` - -Note that many test cases have to be executed as root. - -To run a particular test suite - -``` -$ ./runltp -f syscalls -``` - -To run all tests with `madvise` in the name - -``` -$ ./runltp -f syscalls -s madvise -``` -Also see - -``` -$ ./runltp --help -``` - -Test suites (e.g. syscalls) are defined in the runtest directory. Each file -contains a list of test cases in a simple format, see doc/ltp-run-files.txt. - -Each test case has its own executable or script, these can be executed -directly - -``` -$ testcases/bin/abort01 -``` - -Some have arguments - -``` -$ testcases/bin/mesgq_nstest -m none -``` - -The vast majority of test cases accept the -h (help) switch - -``` -$ testcases/bin/ioctl01 -h -``` - -Many require certain environment variables to be set - -``` -$ LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh -``` - -Most commonly, the path variable needs to be set and also `LTPROOT`, but there -are a number of other variables, `runltp` usually sets these for you. - -Note that all shell scripts need the `PATH` to be set. However this is not -limited to shell scripts, many C based tests need environment variables as -well. - -For more info see `doc/User-Guidelines.asciidoc` or online at -https://github.com/linux-test-project/ltp/wiki/User-Guidelines. - -Network tests -------------- -Network tests require certain setup, described in `testcases/network/README.md` -(online at https://github.com/linux-test-project/ltp/tree/master/testcases/network). - -Containers ----------- - -*Presently running the LTP inside a container is not a shortcut. It -will make things much harder for you.* - -There is a Containerfile which can be used with Docker or -Podman. Currently it can build Alpine and OpenSUSE images. - -The container can be built with a command like: - -``` -$ podman build -t tumbleweed/ltp \ - --build-arg PREFIX=registry.opensuse.org/opensuse/ \ - --build-arg DISTRO_NAME=tumbleweed \ - --build-arg DISTRO_RELEASE=20230925 . -``` - -Or just `podman build .` which will create an Alpine container. - -It contains Kirk in /opt/kirk. So the following will run some tests. - -``` -$ podman run -it --rm tumbleweed/ltp:latest -$ cd /opt/kirk && ./kirk -f ltp -r syscalls -``` - -SUSE also publishes a -[smaller LTP container](https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3D%5Ebenchmark+container%3D.*) -that is not based on the Containerfile. - -Debugging with gdb and strace -============================= - -The new test library runs the actual test, i.e. the `test()` function in a -forked process. To get stack trace of a crashing test in gdb it's needed to -[`set follow-fork-mode child`](https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_25.html). -To trace the test with strace, use strace with `-f` option to enable tracing of forked processes also. - -Developers corner -================= - -Before you start you should read following documents: - -* `doc/Test-Writing-Guidelines.asciidoc` -* `doc/Build-System.asciidoc` -* `doc/LTP-Library-API-Writing-Guidelines.asciidoc` - -There is also a step-by-step tutorial: - -* `doc/C-Test-Case-Tutorial.asciidoc` - -If something is not covered there don't hesitate to ask on the LTP mailing -list. Also note that these documents are available online at: - -* https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines -* https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines -* https://github.com/linux-test-project/ltp/wiki/Build-System -* https://github.com/linux-test-project/ltp/wiki/C-Test-Case-Tutorial - -Although we accept GitHub pull requests, the preferred way is sending patches to our mailing list. - -It's a good idea to test patches on GitHub Actions before posting to mailing -list. Our GitHub Actions setup covers various architectures and distributions in -order to make sure LTP compiles cleanly on most common configurations. -For testing you need to just push your changes to your own LTP fork on GitHub. diff --git a/README.rst b/README.rst new file mode 100644 index 000000000..3e176bd0b --- /dev/null +++ b/README.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Linux Test Project +================== + +Linux Test Project is a joint project started by SGI, OSDL and Bull developed +and maintained by SUSE, Red Hat, Fujitsu, IBM, Cisco, Oracle and others. The +project goal is to deliver tests to the open source community that validate +reliability, robustness, and stability of the Linux Kernel. + +The testing suites contain a collection of tools for testing the Linux kernel +and related features. Our goal is to improve the Linux kernel and system +libraries by bringing test automation. + +.. warning:: + + LTP tests shouldn't run in production systems. In particular, + growfiles, doio, and iogen, stress the I/O capabilities of the systems and + they are intended to find (or cause) problems. + +Some references: + +* `Documentation <http://linux-test-project.rtfd.io/>`_ +* `Source code <https://github.com/linux-test-project/ltp>`_ +* `Releases <https://github.com/linux-test-project/ltp/releases>`_ +* `Mailing List <http://lists.linux.it/listinfo/ltp>`_ +* `Working patches (patchwork) <https://patchwork.ozlabs.org/project/ltp/list/>`_ +* `Working patches (lore.kernel.org) <https://lore.kernel.org/ltp>`_ +* `#ltp @ libera chat <https://libera.chat/>`_ diff --git a/doc/.gitignore b/doc/.gitignore new file mode 100644 index 000000000..173179852 --- /dev/null +++ b/doc/.gitignore @@ -0,0 +1,4 @@ +html/ +build/ +_static/syscalls.rst +syscalls.tbl diff --git a/doc/Makefile b/doc/Makefile index f7e4dd021..e536e95db 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -1,11 +1,8 @@ -# SPDX-License-Identifier: GPL-2.0-or-later -# Copyright (C) 2009, Cisco Systems Inc. -# Ngie Cooper, July 2009 +all: + sphinx-build -b html . html -top_srcdir ?= .. +spelling: + sphinx-build -b spelling -d build/doctree . build/spelling -include $(top_srcdir)/include/mk/env_pre.mk - -RECURSIVE_TARGETS := install - -include $(top_srcdir)/include/mk/generic_trunk_target.mk +clean: + rm -rf html/ diff --git a/doc/_static/custom.css b/doc/_static/custom.css new file mode 100644 index 000000000..cb4d3821d --- /dev/null +++ b/doc/_static/custom.css @@ -0,0 +1,4 @@ +/* set multiline tables cells */ +.wy-table-responsive table td { + white-space: normal; +} diff --git a/doc/conf.py b/doc/conf.py new file mode 100644 index 000000000..fb3e83cf2 --- /dev/null +++ b/doc/conf.py @@ -0,0 +1,201 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +import os +import re +import sphinx +import socket +import urllib.request + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = 'Linux Test Project' +copyright = '2024, Linux Test Project' +author = 'Linux Test Project' +release = '1.0' + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = [ + 'linuxdoc.rstKernelDoc', + 'sphinxcontrib.spelling' +] + +exclude_patterns = ["html*", '_static*'] + +spelling_lang = "en_US" +spelling_warning = True +spelling_exclude_patterns=['users/stats.rst'] +spelling_word_list_filename = "spelling_wordlist" + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = 'sphinx_rtd_theme' +html_static_path = ['_static'] + + +def generate_syscalls_stats(_): + """ + Generate statistics for syscalls. We fetch the syscalls list from the kernel + sources, then we compare it with testcases/kernel/syscalls folder and + generate a file that is included in the statistics documentation section. + """ + output = '_static/syscalls.rst' + + # sometimes checking testcases/kernel/syscalls file names are not enough, + # because in some cases (i.e. io_ring) syscalls are tested, but they are + # part of a more complex scenario. In the following list, we define syscalls + # which we know they are 100% tested already. + white_list = [ + 'rt_sigpending', + 'sethostname', + 'lsetxattr', + 'inotify_add_watch', + 'inotify_rm_watch', + 'newfstatat', + 'pselect6', + 'fanotify_init', + 'fanotify_mark', + 'prlimit64', + 'getdents64', + 'pkey_mprotect', + 'pkey_alloc', + 'pkey_free', + 'io_uring_setup', + 'io_uring_enter', + 'io_uring_register', + 'epoll_pwait2', + 'quotactl_fd', + ] + + # populate with non-syscalls which are present in the kernel sources + # syscalls file + black_list = [ + 'reserved177', + 'reserved193', + 'rseq' + ] + + # fetch syscalls file + error = False + try: + socket.setdefaulttimeout(3) + urllib.request.urlretrieve( + "https://raw.githubusercontent.com/torvalds/linux/master/arch/mips/kernel/syscalls/syscall_n64.tbl", + "syscalls.tbl") + except Exception as err: + error = True + logger = sphinx.util.logging.getLogger(__name__) + msg = "Can't download syscall_n64.tbl from kernel sources" + logger.warning(msg) + + with open(output, 'w+') as stats: + stats.write(f".. warning::\n\n {msg}") + + if error: + return + + text = [ + 'Syscalls\n', + '--------\n\n', + ] + + # collect all available kernel syscalls + regexp = re.compile(r'\d+\s+n64\s+(?P<syscall>\w+)\s+\w+') + ker_syscalls = [] + with open("syscalls.tbl", 'r') as data: + for line in data: + match = regexp.search(line) + if match: + ker_syscalls.append(match.group('syscall')) + + # collect all LTP tested syscalls + ltp_syscalls = [] + for root, _, files in os.walk('../testcases/kernel/syscalls'): + for myfile in files: + if myfile.endswith('.c'): + ltp_syscalls.append(myfile) + + # compare kernel syscalls with LTP tested syscalls + syscalls = {} + for kersc in ker_syscalls: + if kersc in black_list: + continue + + if kersc not in syscalls: + if kersc in white_list: + syscalls[kersc] = True + continue + + syscalls[kersc] = False + + for ltpsc in ltp_syscalls: + if ltpsc.startswith(kersc): + syscalls[kersc] = True + + # generate the statistics file + tested_syscalls = [key for key, val in syscalls.items() if val] + text.append('syscalls which are tested under **testcases/kernel/syscalls**:\n\n') + text.append(f'* kernel syscalls: {len(ker_syscalls)}\n') + text.append(f'* tested syscalls: {len(tested_syscalls)}\n\n') + + # create tested/untested syscalls table + index_tested = 0 + table_tested = [ + 'Tested syscalls\n', + '~~~~~~~~~~~~~~~\n\n', + '.. list-table::\n', + ' :header-rows: 0\n\n', + ] + + index_untest = 0 + table_untest = [ + 'Untested syscalls\n', + '~~~~~~~~~~~~~~~~~\n\n', + '.. list-table::\n', + ' :header-rows: 0\n\n', + ] + + for sysname, tested in syscalls.items(): + if tested: + if (index_tested % 3) == 0: + table_tested.append(f' * - {sysname}\n') + else: + table_tested.append(f' - {sysname}\n') + + index_tested += 1 + else: + if (index_untest % 3) == 0: + table_untest.append(f' * - {sysname}\n') + else: + table_untest.append(f' - {sysname}\n') + + index_untest += 1 + + left = index_tested % 3 + if left > 0: + for index in range(0, left + 1): + table_tested.append(f' -\n') + + left = index_untest % 3 + if left > 0: + for index in range(0, left + 1): + table_untest.append(f' -\n') + + text.extend(table_tested) + text.append('\n') + text.extend(table_untest) + + # write the file + with open(output, 'w+') as stats: + stats.writelines(text) + + +def setup(app): + app.add_css_file('custom.css') + app.connect('builder-inited', generate_syscalls_stats) diff --git a/doc/developers/api_c_tests.rst b/doc/developers/api_c_tests.rst new file mode 100644 index 000000000..adee17f16 --- /dev/null +++ b/doc/developers/api_c_tests.rst @@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +.. Include headers in this file with: +.. .. kernel-doc:: ../../include/tst_test.h + +Developing using C API +====================== diff --git a/doc/developers/api_kvm_tests.rst b/doc/developers/api_kvm_tests.rst new file mode 100644 index 000000000..5ec3fc3a4 --- /dev/null +++ b/doc/developers/api_kvm_tests.rst @@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +.. Include headers in this file with: +.. .. kernel-doc:: ../../include/tst_test.h + +Developing using KVM API +======================== diff --git a/doc/developers/api_network_tests.rst b/doc/developers/api_network_tests.rst new file mode 100644 index 000000000..3e487d7f2 --- /dev/null +++ b/doc/developers/api_network_tests.rst @@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +.. Include headers in this file with: +.. .. kernel-doc:: ../../include/tst_test.h + +Developing using network API +============================ diff --git a/doc/developers/api_shell_tests.rst b/doc/developers/api_shell_tests.rst new file mode 100644 index 000000000..2fb2a5068 --- /dev/null +++ b/doc/developers/api_shell_tests.rst @@ -0,0 +1,4 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Developing using shell API +========================== diff --git a/doc/developers/build_system.rst b/doc/developers/build_system.rst new file mode 100644 index 000000000..1a6ad4c56 --- /dev/null +++ b/doc/developers/build_system.rst @@ -0,0 +1,213 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Build system +============ + +The following document briefly describes the steps and methodologies used for +the new and improved Makefile system. + +The Problem +----------- + +The problem with the old Makefile system is that it was very difficult to +maintain and it lacked any sense of formal structure, thus developing for LTP +and including new targets was more difficult than it should have been +(maintenance). Furthermore, proper option-based cross-compilation was +impossible due to the fact that the Makefiles didn't support a prefixing +system, and the appropriate implicit / static rules hadn't been configured to +compile into multiple object directories for out-of-tree build support (ease of +use / functionality). Finally, there wasn't a means to setup dependencies +between components, such that if a component required ``libltp.a`` in order to +compile, it would go off and compile ``libltp.a`` first (ease of use). + +These items needed to be fixed to reduce maintenance nightmares for the +development community contributing to LTP, and the project maintainers. + +Design +------ + +The system was designed such that including a single GNU Makefile compatible +set in each new directory component is all that's essentially required to +build the system. + +Say you had a directory like the following (with ``.c`` files in them which +directly tie into applications, e.g. baz.c -> baz): + +.. code-block:: + + .../foo/ + |--> Makefile + | + --> bar/ + | + --> Makefile + | + --> baz.c + +.. code-block:: make + :caption: .../foo/Makefile + + # + # Copyright disclaimer goes here -- please use GPLv2. + # + + top_srcdir ?= .. + + include $(top_srcdir)/include/mk/env_pre.mk + include $(top_srcdir)/include/mk/generic_trunk_target.mk + +.. code-block:: make + :caption: .../foo/bar/Makefile + + # + # Copyright disclaimer goes here -- please use GPLv2. + # + + top_srcdir ?= ../.. + + include $(top_srcdir)/include/mk/env_pre.mk + include $(top_srcdir)/include/mk/generic_leaf_target.mk + +Kernel Modules +-------------- + +Some of the tests need to build kernel modules, happily LTP has +infrastructure for this. + +.. code-block:: make + + ifneq ($(KERNELRELEASE),) + + obj-m := module01.o + + else + + top_srcdir ?= ../../../.. + include $(top_srcdir)/include/mk/testcases.mk + + REQ_VERSION_MAJOR := 2 + REQ_VERSION_PATCH := 6 + MAKE_TARGETS := test01 test02 module01.ko + + include $(top_srcdir)/include/mk/module.mk + include $(top_srcdir)/include/mk/generic_leaf_target.mk + + endif + +This is a Makefile example that allows you to build kernel modules inside LTP. +The prerequisites for the build are detected by the ``configure`` script. + +The ``REQ_VERSION_MAJOR`` and ``REQ_VERSION_PATCH`` describe minimal kernel +version for which the build system tries to build the module. + +The build system is also forward compatible with changes in Linux kernel +internal API so that, if module fails to build, the failure is ignored both on +build and installation. If the userspace counterpart of the test fails to load +the module because the file does not exists, the test is skipped. + +Note the ``ifneq($(KERNELRELEASE),)``. The reason it exists, it is that the +Makefile is executed twice: once by LTP build system and once by kernel +kbuild, see ``Documentation/kbuild/modules.rst`` in the Linux kernel tree for +details on external module build. + +Make Rules and Make Variables +----------------------------- + +When using make rules, avoid writing ad hoc rules like: + +.. code-block:: make + + [prog]: [dependencies] + cc -I../../include $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(LDLIBS) \ + -o [prog] [dependencies] + +This makes cross-compilation and determinism difficult, if not impossible. +Besides, implicit rules are your friends and as long as you use ``MAKEOPTS=;`` +in the top-level caller (or do ``$(subst r,$(MAKEOPTS)``) to remove ``-r``), +the compile will complete successfully, assuming all other prerequisites have +been fulfilled (libraries, headers, etc). + +.. list-table:: + :header-rows: 1 + + * - Variable + - Explaination + + * - $(AR) + - The library archiver + + * - $(CC) + - The system C compiler + + * - $(CCP) + - The system C preprocessor + + * - $(CFLAGS) + - C compiler flags + + * - $(CPPFLAGS) + - Preprocessor flags, e.g. ``-I`` arguments + + * - $(DEBUG_CFLAGS) + - Debug flags to pass to ``$(CC)``, ``-g``, etc + + * - $(KVM_LD) + - Special linker for wrapping KVM payload binaries into linkable object + files. Defaults to ``$(LD)``. Change this variable if the KVM Makefile + fails to build files named ``*-payload.o`` + + * - $(LD) + - The system linker (typically ``$(CC)``, but not necessarily) + + * - $(LDFLAGS) + - What to pass in to the linker, including ``-L`` arguments and other ld + arguments, apart from ``-l`` library includes (see ``$(LDLIBS)``). + This should be done in the ``$(CC)`` args passing style when + ``LD := $(CC)``, e.g. ``-Wl,-foo``, as opposed to ``-foo`` + + * - $(LDLIBS) + - Libraries to pass to the linker (e.g. ``-lltp``, etc) + + * - $(LTPLDLIBS) + - LTP internal libraries i.e. these in libs/ directory + + * - $(OPT_CFLAGS) + - Optimization flags to pass into the C compiler, ``-O2``, etc. If you + specify ``-O2`` or higher, you should also specify + ``-fno-strict-aliasing``, because of gcc fstrict-aliasing optimization + bugs in the tree optimizer. Search for **fstrict-aliasing optimization + bug** with your favorite search engine. + + Examples of more recent bugs: tree-optimization/17510 + and tree-optimization/39100. + + Various bugs have occurred in the past due to buggy logic in the + tree-optimization portion of the gcc compiler, from 3.3.x to 4.4. + + * - $(RANLIB) + - What to run after archiving a library + + * - $(WCFLAGS) + - Warning flags to pass to ``$(CC)``, e.g. ``-Werror``, ``-Wall``, etc. + +Make System Variables +--------------------- + +A series of variables are used within the make system that direct what actions +need to be taken. Rather than listing the variables here, please refer to the +comments contained in ``.../include/mk/env_pre.mk``. + +Guidelines and Recommendations +------------------------------ + +Of course, GNU Make manual is the key to understand the Make system, but +following manuals are probably the most important: + +* `Implicit Rules <http://www.gnu.org/software/make/manual/make.html#Implicit-Rules>`_ +* `Variables and Expansion <http://www.gnu.org/software/make/manual/make.html#Using-Variables>`_ +* `Origin Use <http://www.gnu.org/software/make/manual/make.html#Origin-Function>`_ +* `VPath Use <http://www.gnu.org/software/make/manual/make.html#Directory-Search>`_ + +.. warning:: + + Rebuild from scratch before committing anything in the build system. diff --git a/doc/developers/debugging.rst b/doc/developers/debugging.rst new file mode 100644 index 000000000..7a4e6b2ad --- /dev/null +++ b/doc/developers/debugging.rst @@ -0,0 +1,21 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Debugging +========= + +This section explains some tricks which can be used to debug test binaries. + +Debug messages +-------------- + +The LTP framework currently supports ``TDEBUG`` flag test debug messages. These +messages can be enabled using the ``-D`` test's argument. + +Tracing and debugging syscalls +------------------------------ + +The new test library runs the actual test (i.e. the ``test()`` function) in a +forked process. To get stack trace of a crashing test in gdb it's needed to +`set follow-fork-mode child <https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_25.html>`_. +To trace the test, please use ``strace -f`` to enable tracing also for the +forked processes. diff --git a/doc/developers/ltp_library.rst b/doc/developers/ltp_library.rst new file mode 100644 index 000000000..85b69c11c --- /dev/null +++ b/doc/developers/ltp_library.rst @@ -0,0 +1,42 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +LTP Library guidelines +====================== + +General Rules +------------- + +When we extend library API, we need to apply the same general rules that we use +when writing tests, plus: + +#. LTP library tests must go inside ``lib/newlib_tests`` directory +#. LTP documentation has to be updated according to API changes + +Shell API +--------- + +API source code is in ``tst_test.sh``, ``tst_security.sh`` and ``tst_net.sh`` +(all in ``testcases/lib`` directory). + +Changes in the shell API should not introduce uncommon dependencies +(use basic commands installed everywhere by default). + +Shell libraries +~~~~~~~~~~~~~~~ + +Aside from shell API libraries in ``testcases/lib``, it's worth putting +common code for a group of tests into a shell library. The filename +should end with ``_lib.sh`` and the library should load ``tst_test.sh`` or +``tst_net.sh``. + +Shell libraries should have conditional expansion for ``TST_SETUP`` or +``TST_CLEANUP``, to avoid surprises when test specific setup/cleanup function is +redefined by shell library. + +.. code-block:: bash + + # ipsec_lib.sh + # SPDX-License-Identifier: GPL-2.0-or-later + TST_SETUP="${TST_SETUP:-ipsec_lib_setup}" + ... + . tst_test.sh diff --git a/doc/developers/setup_mailinglist.rst b/doc/developers/setup_mailinglist.rst new file mode 100644 index 000000000..1eee0ae9f --- /dev/null +++ b/doc/developers/setup_mailinglist.rst @@ -0,0 +1,50 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Setting up the Mailing list +=========================== + +Before using ``git send-email``, you need to set up your email client to send +emails from the command line. This typically involves configuring an SMTP server +and authentication details. + +Open a terminal and configure Git with your email settings using the following +commands: + +.. code-block:: bash + + git config --global sendemail.from "Your Name <your_email@example.com>" + git config --global sendemail.smtpserver "smtp.example.com" + git config --global sendemail.smtpuser "your_email@example.com" + git config --global sendemail.smtpserverport 587 + git config --global sendemail.smtpencryption tls + +Replace ``smtp.example.com`` with the SMTP server address provided by your email +provider. Replace ``your_email@example.com`` with your email address. Adjust the +SMTP port and encryption settings according to your email provider's +requirements. + +To test the configuration you can use ``--dry-run`` parameter. + +.. code-block:: bash + + git send-email --dry-run --to "ltp@lists.linux.it" --subject "Test Email" --body "This is a test email." HEAD^ + +Depending on your SMTP server's configuration, you may need to authenticate +before sending emails. If required, configure authentication settings using: + +.. code-block:: bash + + git config --global sendemail.smtpuser "your_email@example.com" + git config --global sendemail.smtppass "your_password" + +Replace ``your_email@example.com`` with your email address and ``your_password`` +with your email account password. + +For any corner case, please take a look at the +`email + git <https://git-send-email.io/>`_ documentation. + +.. note:: + + This method still works in most of the cases, but nowadays we often + require to setup a two factor authentication. If this is the case, please + consider setting up Git accordingly. diff --git a/doc/developers/test_case_tutorial.rst b/doc/developers/test_case_tutorial.rst new file mode 100644 index 000000000..ae7d59959 --- /dev/null +++ b/doc/developers/test_case_tutorial.rst @@ -0,0 +1,1038 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Test case tutorial +================== + +This is a step-by-step tutorial on writing a simple C LTP test, where topics +of the LTP and Linux kernel testing will be introduced gradually using a +concrete example. Most sections will include exercises, some trivial and +others not so much. If you find an exercise is leading you off at too much of +a tangent, just leave it for later and move on. + +LTP tests can be written in C or Shell script. This tutorial is **only for tests +written in C** using the new LTP test API. Note that while we go into some +detail on using Git, this is not intended as a canonical or complete guide +for Git. + +Assumptions & Feedback +---------------------- + +We assume the reader is familiar with C, Git and common Unix/Linux/GNU tools +and has some general knowledge of Operating Systems. Experienced Linux +developers may find it too verbose while people new to system level Linux +development may find it overwhelming. + +Comments and feedback are welcome, please direct them to the Mailing list. + +Getting Started +--------------- + +First of all, make sure you have a copy of LTP in the current folder +and we recommended cloning the Linux kernel repository for reference, this +guide will refer to files and directories. + +.. code-block:: bash + + git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git + +There are a number of other repositories which are useful for reference as +well, including the GNU C library ``glibc`` and the alternative C library +``musl``. Some system calls are partially or even entirely implemented in user +land as part of the standard C library. So in these cases, the C library is an +important reference. ``glibc`` is the most common C library for Linux, however +``musl`` is generally easier to understand. + +How system calls are implemented varies from one architecture to another and +across kernel and C library versions. To find out whether a system call is +actually accessing the kernel (whether it is actually a system call) on any +given machine you can use the ``strace`` utility. This intercepts system calls +made by an executable and prints them. We will use this later in the tutorial. + +Choose a System Call to test +---------------------------- + +We will use the ``statx()`` system call, to provide a concrete example of a +test. At the time of writing there is no test for this call which was +introduced in Linux kernel version 4.11. + +Linux system call specific tests are primarily contained in +``testcases/kernel/syscalls``, but you should also ``git grep`` the entire LTP +repository to check for any existing usages of a system call. + +One way to find a system call which is not currently tested by the LTP is to +look at ``include/linux/syscalls.h`` in the kernel tree. + +Something the LTP excels to ensure bug-fixes are back ported to +maintenance releases, so targeting a specific regression is another +option. + +Find an untested System call +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Try to find an untested system call which has a manual page (i.e. ``man +syscall`` produces a result). It is a good idea to Git-clone the latest kernel +man-pages repository. + +.. code-block:: bash + + git clone git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git + +At the time of writing, the difference between the latest man-pages release and +the ``HEAD`` of the repository (usually the latest commit) is well over 100 +commits. This represents about 9 weeks of changes. If you are using a stable +Linux distribution, your man-pages package may well be years old. So as with +the kernel, it is best to have the Git repository as a reference. + +You could also find a system call with untested parameters or use whatever it +is you are planning to use the LTP for. + +Create the test skeleton +------------------------ + +I shall call my test ``statx01.c``, by the time you read this that file name +will probably be taken, so increment the number in the file name as +appropriate or replace ``statx`` with the system call in the chosen exercise. + +.. code-block:: bash + + mkdir testcases/kernel/syscalls/statx + cd testcases/kernel/syscalls/statx + echo statx >> .gitignore + +Next open ``statx01.c`` and add the following boilerplate. Make sure to change +the copyright notice to your name/company, correct the test name and minimum +kernel version if necessary. I will explain what the code does below. + +.. code-block:: c + + // SPDX-License-Identifier: GPL-2.0-or-later + /* + * Copyright (c) 2017 Instruction Ignorer <"can't"@be.bothered.com> + */ + + /*\ + * [Description] + * + * All tests should start with a description of _what_ we are testing. + * Non-trivial explanations of _how_ the code works should also go here. + * Include relevant links, Git commit hashes and CVE numbers. + * Inline comments should be avoided. + */ + + #include "tst_test.h" + + static void run(void) + { + tst_res(TPASS, "Doing hardly anything is easy"); + } + + static struct tst_test test = { + .test_all = run, + .min_kver = "4.11", + }; + +Starting with the ``#include`` statement we copy in the main LTP test library +headers. This includes the most common test API functions and the test harness +initialisation code. It is important to note that this is a completely +ordinary, independent C program, however ``main()`` is missing because it is +implemented in ``tst_test.h``. + +We specify what code we want to run as part of the test using the ``tst_test +test`` structure. Various callbacks can be set by the test writer, including +``test.test_all``, which we have set to ``run()``. The test harness will execute +this callback in a separate process (using ``fork()``), forcibly terminating it +if it does not return after ``test.timeout`` seconds. + +We have also set ``test.min_kver`` to the kernel version where ``statx`` was +introduced. The test library will determine the kernel version at runtime. If +the version is less than 4.11 then the test harness will return ``TCONF``, +indicating that this test is not suitable for the current system +configuration. + +Occasionally features are back ported to older kernel versions, so ``statx`` may +exist on kernels with a lower version. However we don't need to worry about +that unless there is evidence of it happening. + +As mentioned in the code itself, you should specify what you are testing and +the expected outcome, even if it is relatively simple. If your program flow is +necessarily complex and difficult to understand (which is often the case when +trying to manipulate the kernel into doing something bad), then a detailed +explanation of how the code works is welcome. + +What you should not do, is use inline comments or include the same level of +explanation which is written here. As a general rule, if something is easy to +document, then the code should also be easy to read. So don't document the easy +stuff (except for the basic test specification). + +Before continuing we should compile this and check that the basics work. In +order to compile the test we need a ``Makefile`` in the same subdirectory. If +one already exists, then nothing needs to be done, otherwise add one with the +following contents. + +.. code-block:: make + + # SPDX-License-Identifier: GPL-2.0-or-later + # Copyright (c) 2019 Linux Test Project + + top_srcdir ?= ../../../.. + + include $(top_srcdir)/include/mk/testcases.mk + + include $(top_srcdir)/include/mk/generic_leaf_target.mk + +This will automatically add ``statx01.c`` as a build target producing a +``statx01`` executable. Unless you have heavily deviated from the tutorial, and +probably need to change ``top_srcdir``, nothing else needs to be done. + +Normally, if you were starting a Makefile from scratch, then you would need to +add ``statx01`` as a build target. Specifying that you would like to run some +program (e.g. ``gcc`` or ``clang``) to transform ``statx01.c`` into ``statx01``. +Here we don't need to do that, but sometimes it is still necessary. For example, +if we needed to link to the POSIX threading library, then we could add the +following line after ``testcases.mk``. + +.. code-block:: make + + statx01: CFLAGS += -pthread + +Assuming you are in the test's subdirectory ``testcases/kernel/syscalls/statx``, +please do: + +.. code-block:: bash + + make + ./statx01 + +This should build the test and then run it. However, even though the test is +in the ``syscalls`` directory it won't be automatically ran as part of the +_syscalls_ test group (remember ``./runltp -f syscalls`` from the +``README.md``?). For this we need to add it to the ``runtest`` file. So open +``runtest/syscalls`` and add the lines starting with a ``+``. + +.. code-block:: + + statvfs01 statvfs01 + statvfs02 statvfs02 + + +statx01 statx01 + + + stime01 stime01 + stime02 stime02 + +The ``runtest`` files are in a two column format. The first column is the test +name, which is mainly used by test runners for reporting and filtering. It is +just a single string of text with no spaces. The second column, which can +contain spaces, is passed to the shell in order to execute the test. Often it +is just the executable name, but some tests also take arguments (the LTP has a +library for argument parsing, by the way). + +If you haven't done so already, we should add all these new files to Git. It +is vitally important that you do not make changes to the master branch. If you +do then pulling changes from upstream becomes a major issue. So first of all +create a new branch. + +.. code-block:: bash + + git checkout -b statx01 master + +Now we want to add the files we have created or modified, but before doing a +commit make sure you have configured Git correctly. You need to at least set +your Name and e-mail address in ``~/.gitconfig``, but there are some other +settings which come in handy too. My relatively simple configuration is similar +to the below: + +.. code-block:: ini + + [user] + name = Sarah Jane + email = sjane@e-mail.address + [core] + editor = emacs + [sendemail] + smtpServer = smtp.server.address + +Obviously you need to at least change your name and e-mail. The SMTP server is +useful for ``git send-email``, which we will discuss later. The editor value is +used for things like writing commits (without the ``-m`` option). + +.. code-block:: bash + + git add -v :/testcases/kernel/syscalls/statx :/runtest/syscalls + git commit -m "statx01: Add new test for statx syscall" + +This should add all the new files in the ``statx`` directory and the ``runtest`` +file. It is good practice to commit early and often. Later on we will do a +Git-rebase, which allows us to clean up the commit history. So don't worry +about how presentable your commit log is for now. Also don't hesitate to +create a new branch when doing the exercises or experimenting. This will allow +you to diverge from the tutorial and then easily come back again. + +I can't emphasize enough that Git makes things easy through branching and that +things quickly get complicated if you don't do it. However if you do get into +a mess, Git-reflog and Git-reset, will usually get you out of it. If you also +mess that up then it may be possible to cherry pick 'dangling' commits out of +the database into a branch. + +Report TCONF instead of TPASS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Maybe the test should report ``TCONF: Not implemented`` instead or perhaps +``TBROK``. Try changing it do so. + +Check Git ignores the executable +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Is your ``.gitignore`` correct? + +Run make check +~~~~~~~~~~~~~~~~~~ + +Check coding style with ``make check``. + +Install the LTP and run the test with runtest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Run ``statx01`` on its own, also using ``-I0`` amd ``-I10``. + +Call the system call +-------------------- + +At the time of writing ``statx`` has no ``glibc`` wrapper. It is also fairly common +for a distribution's C library version to be older than its kernel or it may use a +cut down C library in comparison to the GNU one. So we must call ``statx()`` +using the general ``syscall()`` interface. + +The LTP contains a library for dealing with the ``syscall`` interface, which is +located in ``include/lapi``. System call numbers are listed against the relevant +call in the ``*.in`` files (e.g. ``x86_64.in``) which are used to generate +``syscalls.h``, which is the header you should include. On rare occasions you +may find the system call number is missing from the ``*.in`` files and will need +to add it (see ``include/lapi/syscalls/strip_syscall.awk``). + +System call numbers vary between architectures, hence there are multiple +``*.in`` files for each architecture. You can find the various values for the +``statx`` system call across a number of ``unistd.h`` files in the Linux kernel. + +Note that we don't use the system-call-identifier value available in +``/usr/include/linux/uinstd.h`` because the kernel might be much newer than the +user land development packages. + +For ``statx`` we had to add ``statx 332`` to ``include/lapi/syscalls/x86_64.in``, +``statx 383`` to ``include/lapi/syscalls/powerpc.in``, etc. Now lets look at +the code, which I will explain in more detail further down. + +.. code-block:: c + + /* + * Test statx + * + * Check if statx exists and what error code it returns when we give it dodgy + * data. + */ + + #include <stdint.h> + #include "tst_test.h" + #include "lapi/syscalls.h" + + struct statx_timestamp { + int64_t tv_sec; + uint32_t tv_nsec; + int32_t __reserved; + }; + + struct statx { + uint32_t stx_mask; + uint32_t stx_blksize; + uint64_t stx_attributes; + uint32_t stx_nlink; + uint32_t stx_uid; + uint32_t stx_gid; + uint16_t stx_mode; + uint16_t __spare0[1]; + uint64_t stx_ino; + uint64_t stx_size; + uint64_t stx_blocks; + uint64_t stx_attributes_mask; + struct statx_timestamp stx_atime; + struct statx_timestamp stx_btime; + struct statx_timestamp stx_ctime; + struct statx_timestamp stx_mtime; + uint32_t stx_rdev_major; + uint32_t stx_rdev_minor; + uint32_t stx_dev_major; + uint32_t stx_dev_minor; + uint64_t __spare2[14]; + }; + + static int sys_statx(int dirfd, const char *pathname, int flags, + unsigned int mask, struct statx *statxbuf) + { + return tst_syscall(__NR_statx, dirfd, pathname, flags, mask, statxbuf); + } + + ... + +So the top part of the code is now boiler plate for calling ``statx``. It is +common for the kernel to be newer than the user land libraries and headers. So +for new system calls like ``statx``, we copy, with a few modifications, the +relevant definitions into the LTP. This is somewhat like 'vendoring', although +we are usually just copying headers required for interacting with the Kernel's +ABI (Application Binary Interface), rather than internalising actual +functionality. + +So from the top we include the ``stdint.h`` library which gives us the standard +``(u)int*_t`` type definitions. We use these in place of the Kernel type +definitions such as ``__u64`` in ``linux/types.h``. We then have a couple of +structure definitions which form part of the ``statx`` API. These were copied +from ``include/uapi/linux/stat.h`` in the Kernel tree. + +After that, there is a wrapper function, which saves us from writing +``tst_syscall(__NR_statx, ...``, every time we want to make a call to +``statx``. This also provides a stub for when ``statx`` is eventually integrated +into the LTP library and also implemented by the C library. At that point we +can switch to using the C library implementation if available or fallback to +our own. + +The advantage of using the C library implementation is that it will often be +better supported across multiple architectures. It will also mean we are using +the system call in the same way most real programs would. Sometimes there are +advantages to bypassing the C library, but in general it should not be our +first choice. + +The final test should do a check during configuration (i.e. when we run +``./configure`` before building) which checks if the ``statx`` system call and +associated structures exists. This requires writing an ``m4`` file for use with +``configure.ac`` which is processed during ``make autotools`` and produces the +configure script. + +For the time being though we shall just ignore this. All you need to know for +now is that this is a problem which eventually needs to be dealt with and that +there is a system in place to handle it. + +.. code-block:: c + + ... + + static void run(void) + { + struct statx statxbuf = { 0 }; + + TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); + + if (TST_RET == 0) + tst_res(TFAIL, "statx thinks it can stat NULL"); + else if (TST_ERR == EFAULT) + tst_res(TPASS, "statx set errno to EFAULT as expected"); + else + tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); + } + + static struct tst_test test = { + .test_all = run, + .min_kver = "4.11", + }; + +The ``TEST`` macro sets ``TST_RET`` to the return value of ``tst_statx()`` and +``TST_ERR`` to the value of ``errno`` immediately after the functions +return. This is mainly just for convenience, although it potentially could +have other uses. + +We check whether the return value indicates success and if it doesn't also +check the value of ``errno``. The last call to ``tst_res`` includes ``TERRNO``, +which will print the current error number and associated description in +addition to the message we have provided. Note that it uses the current value +of ``errno`` not ``TST_ERR``. + +What we should have done in the example above is use ``TTERRNO`` which takes the +value of ``TST_ERR``. + +If we try to run the test on a kernel where ``statx`` does not exist, then +``tst_syscall`` will cause it to fail gracefully with ``TCONF``. Where ``TCONF`` +indicates the test is not applicable to our configuration. + +The function ``tst_syscall`` calls ``tst_brk(TCONF,...)`` on failure. ``tst_brk`` +causes the test to exit immediately, which prevents any further test code from +being run. + +What are the differences between ``tst_brk`` and ``tst_res``? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See ``include/tst_test.h``. Also what do they have in common? + +What happens if you call ``tst_res(TINFO, ...)`` after ``sys_statx``? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Does the test still function correctly? + +Extend the test to handle other basic error conditions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For example, see if you can trigger ``ENOENT`` instead. You shouldn't +have to create any files, which is discussed in the next section. + +Setup, Cleanup and files +------------------------ + +Some tests require resources to be allocated, or system settings to be +changed, before the test begins. This ``setup`` only has to be done once at the +beginning and at the end of the test needs to be removed or reverted. The +``cleanup`` also has to be done regardless of whether the test breaks. + +Fortunately, like most test libraries, we have setup and cleanup (teardown) +callbacks. ``setup`` is called once before ``run`` and ``cleanup`` is called once +afterwards. Note that ``run`` itself can be called multiple times by the test +harness, but that ``setup`` and ``cleanup`` are only called once. + +If either your code, a ``SAFE_*`` macro or a library function such as +``tst_syscall`` call ``tst_brk``, then ``run`` will exit immediately and the +``cleanup`` function is then called. Once ``cleanup`` is completed, the test +executable will then exit altogether abandoning any remaining iterations of +``run``. + +For ``statx`` we would like to create some files or file like objects which we +have control over. Deciding where to create the files is easy, we just create +it in the current working directory and let the LTP test harness handle where +that should be by setting ``.needs_tmpdir = 1``. + +.. code-block:: c + + /* + * Test statx + * + * Check if statx exists and what error code it returns when we give it dodgy + * data. Then stat a file and check it returns success. + */ + + #include <stdint.h> + #include "tst_test.h" + #include "lapi/syscalls.h" + #include "lapi/fcntl.h" + + #define FNAME "file_to_stat" + #define STATX_BASIC_STATS 0x000007ffU + + /*************** statx structure and wrapper goes here ! ***************/ + ... + +We have added an extra include ``lapi/fcntl.h`` which wraps the system header by +the same name (``#include <fcntl.h>``). This header ensures we have definitions +for recently added macros such as ``AT_FDCWD`` by providing fall backs if the +system header does not have them. The ``lapi`` directory contains a number of +headers like this. + +At some point we may wish to add ``lapi/stat.h`` to provide a fall back for +macros such as ``STATX_BASIC_STATS``. However for the time being we have just +defined it in the test. + + +.. code-block:: c + + ... + + static void setup(void) + { + SAFE_TOUCH(FNAME, 0777, NULL); + } + + static void run(void) + { + struct statx statxbuf = { 0 }; + + TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); + if (TST_RET == 0) + tst_res(TFAIL, "statx thinks it can stat NULL"); + else if (TST_ERR == EFAULT) + tst_res(TPASS, "statx set errno to EFAULT as expected"); + else + tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); + + TEST(sys_statx(AT_FDCWD, FNAME, 0, STATX_BASIC_STATS, &statxbuf)); + if (TST_RET == 0) + tst_res(TPASS, "It returned zero so it must have worked!"); + else + tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); + } + + static struct tst_test test = { + .setup = setup, + .test_all = run, + .min_kver = "4.11", + .needs_tmpdir = 1 + }; + +The ``setup`` callback uses one of the LTP's ``SAFE`` functions to create an empty +file ``file_to_stat``. Because we have set ``.needs_tmpdir``, we can just create +this file in the present working directory. We don't need to create a +``cleanup`` callback yet because the LTP test harness will recursively delete +the temporary directory and its contents. + +The ``run`` function can be called multiple times by the test harness, however +``setup`` and ``cleanup`` callbacks will only be ran once. + +.. warning:: + + By this point you may have begun to explore the LTP library headers or older + tests. In which case you will have come across functions from the old API such + as ``tst_brkm``. The old API is being phased out, so you should not use these + functions. + +So far we haven't had to do any clean up. So our example doesn't answer the +question "what happens if part of the clean up fails?". To answer this we are +going to modify the test to ask the (highly contrived) question "What happens +if I create and open a file, then create a hard-link to it, then call open +again on the hard-link, then ``stat`` the file". + + +.. code-block:: c + + #define LNAME "file_to_stat_link" + + ... + + static void setup(void) + { + fd = SAFE_OPEN(FNAME, O_CREAT, 0777); + SAFE_LINK(FNAME, LNAME); + lfd = SAFE_OPEN(LNAME, 0); + } + + static void cleanup(void) + { + if (lfd != 0) + SAFE_CLOSE(lfd); + + if (fd != 0) + SAFE_CLOSE(fd); + } + + static void run(void) + { + ... + + TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); + if (TST_RET == 0) + tst_res(TPASS, "It returned zero so it must have worked!"); + else + tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); + } + + static struct tst_test test = { + .setup = setup, + .cleanup = cleanup, + .test_all = run, + .tcnt = 2, + .min_kver = "4.11", + .needs_tmpdir = 1 + }; + +Because we are now opening a file, we need a ``cleanup`` function to close the +file descriptors. We have to manually close the files to ensure the temporary +directory is deleted by the test harness (see the +https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[test +writing guidelines] for details). + +As a matter of good practice, the file descriptors are closed in reverse +order. In some circumstances the order in which clean up is performed is +significant. In that case resources created towards the end of 'setup' are +dependent on ones near the beginning. So during 'cleanup' we remove the +dependants before their dependencies. + +If, for some reason, the file descriptor ``lfd`` became invalid during the test, +but ``fd`` was still open, we do not want ``SAFE_CLOSE(lfd)`` to cause the +``cleanup`` function to exit prematurely. If it did, then ``fd`` would remain open +which would cause problems on some file systems. + +Nor do we want to call ``cleanup`` recursively. So during 'cleanup' ``tst_brk``, +and consequently the ``SAFE`` functions, do not cause the test to exit with +``TBROK``. Instead they just print an error message with ``TWARN``. + +It is not entirely necessary to check if the file descriptors have a none zero +value before attempting to close them. However it avoids a bunch of spurious +warning messages if we fail to open ``file_to_stat``. Test case failures can be +difficult to interpret at the best of times, so avoid filling the log with +noise. + +Check ``statx`` returns the correct number of hard links +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The field ``statx.stx_nlink`` should be equal to 2, right? + +Git-branch +~~~~~~~~~~ + +We are about to make some organisational changes to the test, so now would be +a good time to branch. Then we can switch between the old and new versions, to +check the behavior has not been changed by accident. + +Split the test +-------------- + +In our current test, we have essentially rolled two different test cases into +one. Firstly we check if an error is returned when bad arguments are provided +and secondly we check what happens when we stat an actual file. Quite often it +makes sense to call ``tst_res`` multiple times in a single test case because we +are checking different properties of the same result, but here we are clearly +testing two different scenarios. + +So we should split the test in two. One obvious way to do this is to create +``statx02.c``, but that seems like overkill in order to separate two simple test +cases. So, for now at least, we are going to do it a different way. + +.. code-block:: c + + ... + + static void run_stat_null(void) + { + struct statx statxbuf = { 0 }; + + TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); + if (TST_RET == 0) + tst_res(TFAIL, "statx thinks it can stat NULL"); + else if (TST_ERR == EFAULT) + tst_res(TPASS, "statx set errno to EFAULT as expected"); + else + tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); + } + + static void run_stat_symlink(void) + { + struct statx statxbuf = { 0 }; + + TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); + if (TST_RET == 0) + tst_res(TPASS, "It returned zero so it must have worked!"); + else + tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); + } + + static void run(unsigned int i) + { + switch(i) { + case 0: run_stat_null(); + case 1: run_stat_symlink(); + } + } + + static struct tst_test test = { + .setup = setup, + .cleanup = cleanup, + .test = run, + .tcnt = 2, + .min_kver = "4.11", + .needs_tmpdir = 1 + }; + +So we have used an alternative form of the ``test`` or ``run`` callback which +accepts an index. Some tests use this index with an array of parameters and +expected return values. Others do something similar to the above. The index +can be used how you want so long as each iteration calls ``tst_res`` in a +meaningful way. + +If an iteration fails to return a result (i.e. call ``tst_res`` with a value +other than ``TINFO``) then the test harness will report ``TBROK`` and print the +iteration which failed. This prevents a scenario in your test from silently +failing due to some faulty logic. + +What is wrong with the switch statement? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Were you paying attention? Also see the output of ``make check``. + +Test a feature unique to statx +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +So far we have not tested anything which is unique to ``statx``. So, for +example, you could check stx_btime is correct (possibly only to within a +margin of error) and that it differs from ``stx_mtime`` after writing to the +file. + +Alternatively you could check that ``stx_dev_major`` and ``stx_dev_minor`` are set +correctly. Note that the LTP has helper functions for creating devices and +file systems. + +This could be quite a challenging exercise. You may wish to tackle an +altogether different test scenario instead. If you get stuck just move onto +the next section and come back later. + +Submitting the test for review +------------------------------ + +Ignoring the fact we should probably create ``lapi/stat.h`` along with a bunch +of fallback logic in the build system. We can now get our test ready for +submission. + +The first thing you need to do before considering submitting your test is run +``make check-statx01`` or ``make check`` in the test's directory. Again, we use +the kernel style guidelines where possible. Next you should create a new +branch, this will allow you to reshape your commit history without fear. + +After that we have the pleasure of doing an interactive ``rebase`` to clean up +our commit history. In its current form the test only really needs a single +commit, but if you have been using Git correctly then you should have +many. The main reason we want to compress it to a single commit, is to make +the LTP's Git-log readable. It also allows us to write a coherent description +of the work as a whole in retrospective. Although, when adding a new test, the +test description in the code will probably make the commit message redundant. + +Anyway, as an example, we shall look at my personal commit history from this +tutorial and ``rebase`` it. You should try following along with your own +repository. First lets look at the commit history since we branched from +master. + +.. code-block:: bash + + git log --oneline master..HEAD + 152d39fe7 (HEAD -> tutorial-rebase2, tutorial-rebase) tutorial: Start Submitting patch section + 70f7ce7ce statx01: Stop checkpatch from complaining + bb0332bd7 tutorial: Fix review problems + 6a87a084a statx01: Fix review problems + d784b1e85 test-writing-guidelines: Remove old API argument + c26e1be7a fixup! tutorial + 1e24a5fb5 (me/tutorial-rebase) fixup! tutorial + 568a3f7be fixup! tutorial + 09dd2c829 statx: stage 6 + bfeef7902 statx: stage 5b + 76e03d714 statx: stage 5a + 98f5bc7ac statx: stage 4 + 6f8c16438 statx: stage 3 (Add statx01) + 5d93b84d8 Add statx and other syscall numbers + 5ca627b78 tutorial: Add a step-by-step C test tutorial + +So we have told git to show all the commits which don't exist in ``master``, but +are in ``HEAD``, where ``HEAD`` is the top of the current branch. The current +branch is ``tutorial-rebase2`` which I just created. I have already done one +``rebase`` and submitted a patch for review, so my original branch was just called +``tutorial``. + +As usual my commit history is starting to look like a bit of mess! There is +even a commit in there which should not be in the this branch (Remove old API +argument), however it can be ignored for now and 'cherry picked' into a new branch +later. + +For my patch I actually need at least two commits, one which contains the +tutorial text and one which contains the test and associated files. So first +of all I want to 'squash' (amalgamate) all the commits appended with +``tutorial:`` into the bottom commit. + +.. code-block:: bash + + $ git rebase -i 5ca627b78\^ + ... + +This begins an interactive ``rebase`` where commit ``5ca6427b78`` is the earliest +commit we want to edit. The ``^`` symbol after the commit hash, specifies the +commit before this one. The interactive ``rebase`` command takes the last commit +we want to keep unaltered as it's argument (in other words it takes a +non-inclusive range). + +Upon entering a similar command you will be presented with a text file +similar to the following. The file should be displayed in your text editor of +choice, if it doesn't, then you may change the editor variable in +``.gitconfig``. + +.. code-block:: bash + + pick 5ca627b78 tutorial: Add a step-by-step C test tutorial + pick 5d93b84d8 Add statx and other syscall numbers + pick 6f8c16438 statx: stage 3 (Add statx01) + pick 98f5bc7ac statx: stage 4 + pick 76e03d714 statx: stage 5a + pick bfeef7902 statx: stage 5b + pick 09dd2c829 statx: stage 6 + pick 568a3f7be fixup! tutorial + pick 1e24a5fb5 fixup! tutorial + pick c26e1be7a fixup! tutorial + pick d784b1e85 test-writing-guidelines: Remove old API argument + pick 6a87a084a statx01: Fix review problems + pick bb0332bd7 tutorial: Fix review problems + pick 70f7ce7ce statx01: Stop checkpatch from complaining + pick 152d39fe7 tutorial: Start Submitting patch section + +The last commit from Git-log is shown at the top. The left hand column +contains the commands we want to run on each commit. ``pick`` just means we +re-apply the commit as-is. We can reorder the lines to apply the commits in a +different order, but we need to be careful when reordering commits to the same +file. If your ``rebase`` results in a merge conflict, then you have probably +reordered some commits which contained changes to the same piece of code. + +Perhaps a better name for the interactive ``rebase`` command would be 'replay'. As +we pick a point in the commit history, undo all those commits before that +point, then reapply them one at a time. During the replay we can reorder the +commits, drop, merge, split and edit them, creating a new history. + +The commands I am going to use are ``reword`` and ``fixup``. The ``reword`` command +allows you to edit a single commit's message. The 'fixup' command 'squashes' a +commit into the commit above/preceding it, merging the two commits into +one. The commit which has ``fixup`` applied has its commit message deleted. If +you think a commit might have something useful in its message then you can use +``squash`` instead. + +.. code-block:: bash + + reword 5ca627b78 tutorial: Add a step-by-step C test tutorial + fixup 568a3f7be fixup! tutorial + fixup 1e24a5fb5 fixup! tutorial + fixup c26e1be7a fixup! tutorial + fixup bb0332bd7 tutorial: Fix review problems + fixup 152d39fe7 tutorial: Start Submitting patch section + fixup 276edecab tutorial: Save changes before rebase + pick 5d93b84d8 Add statx and other syscall numbers + pick 6f8c16438 statx: stage 3 (Add statx01) + pick 98f5bc7ac statx: stage 4 + pick 76e03d714 statx: stage 5a + pick bfeef7902 statx: stage 5b + pick 09dd2c829 statx: stage 6 + pick d784b1e85 test-writing-guidelines: Remove old API argument + pick 6a87a084a statx01: Fix review problems + +So all the commits marked with ``fixup`` will be re-played by Git immediately +after 5ca62 at the top. A new commit will then be created with the amalgamated +changes of all the commits and 5ca62's log message. It turns out that I didn't +need to reword anything, but there is no harm in checking. It is easy to +forget the ``Signed-off-by:`` line. + +I could now do the same for the commits to the ``statx`` test, making the commit +message prefixes consistent. However I am not actually going to submit the +test (yet). + +I won't attempt to show you this, but if you need to do the opposite and split +apart a commit. It is also possible using Git-rebase by marking a line with +``edit``. This will pause Git just after replaying the marked commit. You can +then use a 'soft' Git-reset to bring the selected commit's changes back into +the 'index' where you are then able to un-stage some parts before +re-committing. + +You can also use ``edit`` and ``git commit --amend`` together to change a commit +deep in your history, but without resetting the 'index'. The 'index' contains +changes which you have staged with ``git add``, but not yet committed. + +So now that the commit history has been cleaned up, we need to submit a patch +to the mailing list or make a pull request on GitHub. The mailing list is the +preferred place to make submissions and is more difficult for most people, so +I will only cover that method. + +Just before we create the patch, we need to check that our changes will still +apply to the master branch without problems. To do this we can use another +type of ``rebase`` and then try rebuilding and running the test. + +.. code-block:: bash + + git checkout master + git pull origin + git checkout tutorial-rebase2 + git rebase master + +Above, I update the master branch and then replay our changes onto it using +``git rebase master``. You may find that after the rebase there is a merge +conflict. This will result in something which looks like the following (taken +from a Makefile conflict which was caused by reordering commits in a ``rebase``). + +.. code-block:: diff + + <<<<<<< HEAD + cve-2016-7117: LDFLAGS += -lpthread + ======= + cve-2014-0196: LDFLAGS += -lpthread -lutil -lrt + cve-2016-7117: LDFLAGS += -lpthread -lrt + >>>>>>> 4dbfb8e79... Add -lrt + +The first line tells us this is the beginning of a conflict. The third line +separates the two conflicting pieces of content and the last line is the end +of the conflict. Usually, all you need to do is remove the lines you don't +want, stage the changes and continue the ``rebase`` with ``git rebase +--continue``. + +In order to create a patch e-mail we use +`git format-patch <https://git-scm.com/docs/git-format-patch>`_, +we can then send that e-mail using +`git send-email <https://git-scm.com/docs/git-send-email>`_. +It is also possible to import the patch (``mbox``) file into a number of e-mail +programs. + +.. code-block:: bash + + $ git format-patch -1 -v 2 -o output --to ltp@lists.linux.it fd3cc8596 + output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch + +The first argument ``-1`` specifies we want one commit from fd3cc8596 +onwards. If we wanted this commit and the one after it we could specify ``-2`` +instead. + +This is my second patch submission so I have used ``-v 2``, which indicates this +is the second version of a patch set. The ``-o`` option specifies the output +directory (literally called ``output``). The ``--to`` option adds the ``To:`` e-mail +header, which I have set to the LTP mailing list. + +We can then send this patch with the following command sans ``--dry-run``. + +.. code-block:: bash + + git send-email --dry-run output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch + +Git will ask some questions (which you can ignore) and then tell you what it +would do if this weren't a dry-run. In order for this to work you have to have +a valid SMTP server set in ``.gitconfig`` and also be signed up to the LTP +mailing list under the same e-mail address you have configured in Git. You can +sign up at https://lists.linux.it/listinfo/ltp. + +Doing code review +----------------- + +While waiting for your test to be reviewed, you are invited and encouraged to +review other contributors' code. This may seem bizarre when you are completely +new to the project, but there are two important ways in which you can +contribute here: + +A. Point out logical errors in the code. +B. Improve your own understanding + +It doesn't matter whether you know the canonical way of writing an LTP test in +C. An error of logic, when properly explained, is usually indisputable. These +are the most important errors to find as they always result in false test +results. Once someone points out such an error it is usually obvious to +everyone that it is a bug and needs to be fixed. + +Obviously testing the patch is one way of finding errors. You can apply +patches using ``git am``. Then it is just a case of compiling and running the +tests. + +Finally, reading and attempting to comment on other peoples patches, gives +you a better understanding of the reviewers perspective. This is better for +the project and for you. + +Style and organisational issues are best left to after you have found logical +errors. + +Final notes +----------- + +Hopefully you can now grasp the structure of an LTP test and have some idea of +what is available in the LTP test library. There are a vast number of library +functions available (mainly located in include and lib), some of which are +documented in the test writing guidelines and many of which are not. + +We have only scratched the surface of the immense technical complexity of +systems programming across multiple Kernel and C lib versions as well as +different hardware architectures. The important thing to take away from this +is that you have to be conscientious of what will happen on systems different +from yours. The LTP has a huge and varied user base, so situations you may +think are unlikely can and do happen to somebody. + +Of course you don't want to spend time allowing for situations which may never +arise either, so you have to do your research and think about each situation +critically. The more systems you can test on before submitting your changes, +the better, although we understand not everyone has access to a lab. + +One important topic which has not been covered by this tutorial, is +multi-process or multi-threaded testing. The LTP library functions work inside +child processes and threads, but their semantics change slightly. There are +also various helper functions for synchronising and forking processes. + +.. note:: + + When it comes time to submit a test, the preferred way to do it is on the + mailing list although you can also use GitHub. The LTP follows similar rules + to the kernel for formatting and submitting patches. Generally speaking the + review cycle is easier for small patches, so try to make small changes or + additions where possible. diff --git a/doc/developers/writing_tests.rst b/doc/developers/writing_tests.rst new file mode 100644 index 000000000..7b4b70bd0 --- /dev/null +++ b/doc/developers/writing_tests.rst @@ -0,0 +1,537 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Writing tests +============= + +This document describes LTP guidelines and it's intended for anybody who wants +to write or to modify a LTP testcase. It's not a definitive guide and it's not, +by any means, a substitute for common sense. + +Guide to clean and understandable code +-------------------------------------- + +Testcases require that the source code is easy to follow. When a test starts to +fail, the failure has to be analyzed and clean test codebase makes this task +much easier and quicker. + +Keep things simple +~~~~~~~~~~~~~~~~~~ + +It's worth to keep testcases simple or, better, as simple as possible. + +The kernel and libc are tricky beasts and the complexity imposed by their +interfaces is quite high. Concentrate on the interface you want to test and +follow the UNIX philosophy. + +It's a good idea to make the test as self-contained as possible too, ideally +tests should not depend on tools or libraries that are not widely available. + +Do not reinvent the wheel! + +* Use LTP standard interface +* Do not add custom PASS/FAIL reporting functions +* Do not write Makefiles from scratch, use LTP build system instead +* Etc. + +Keep functions and variable names short +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Choosing a good name for an API functions or even variables is a difficult +task do not underestimate it. + +There are a couple of customary names for different things that help people to +understand code, for example: + +* For loop variables are usually named with a single letter ``i``, ``j``, ... +* File descriptors ``fd`` or ``fd_foo``. +* Number of bytes stored in file are usually named as ``size`` or ``len`` +* Etc. + +Do not overcomment +~~~~~~~~~~~~~~~~~~ + +Comments can sometimes save your day, but they can easily do more harm than +good. There has been several cases where comments and actual implementation +drifted slowly apart which yielded into API misuses and hard to find bugs. +Remember there is only one thing worse than no documentation: wrong +documentation. + +Ideally, everybody should write code that is obvious, which unfortunately isn't +always possible. If there is a code that requires to be commented, keep it +short and to the point. These comments should explain *why* and not *how* +things are done. + +Never ever comment the obvious. + +In case of LTP testcases, it's customary to add an asciidoc formatted comment +paragraph with highlevel test description at the beginning of the file right +under the GPL SPDX header. This helps other people to understand the overall +goal of the test before they dive into the technical details. It's also +exported into generated documentation hence it should mostly explain what is +tested. + +DRY (Code duplication) +~~~~~~~~~~~~~~~~~~~~~~ + +Copy & paste is a good servant but very poor master. If you are about to copy a +large part of the code from one testcase to another, think what would happen if +you find bug in the code that has been copied all around the tree. What about +moving it to a library instead? + +The same goes for short but complicated parts, whenever you are about to copy & +paste a syscall wrapper that packs arguments accordingly to machine +architecture or similarly complicated code, put it into a header instead. + +C coding style +-------------- + +LTP adopted `Linux kernel coding style <https://www.kernel.org/doc/html/latest/process/coding-style.html>`_. +Run ``make check`` in the test's directory and/or use ``make check-$TCID``, it +uses (among other checks) our vendored version of +`checkpatch.pl <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl>`_ +script from kernel git tree. + +.. note:: + If ``make check`` does not report any problems, the code still may be wrong + as all tools used for checking only look for common mistakes. + +The following linting code can be found when we run ``make check``: + +.. list-table:: + :header-rows: 1 + + * - Linting code + - Message + - Explanation + + * - LTP-001 + - Library source files have ``tst_`` prefix + - API source code is inside headers in ``include/{empty}*.h``, + ``include/lapi/{empty}*.h`` (backward compatibility for old kernel and + libc) and C sources in ``lib/{empty}*.c``. Files must have ``tst_`` + prefix. + + * - LTP-002 + - ``TST_RET`` and ``TST_ERR`` are never modified by test library functions + - The test author is guaranteed that the test API will not modify these + variables. This prevents silent errors where the return value and + errno are overwritten before the test has chance to check them. + + The macros which are clearly intended to update these variables. That + is ``TEST`` and those in ``tst_test_macros.h``. Are of course allowed to + update these variables. + + * - LTP-003 + - Externally visible library symbols have the ``tst_`` prefix + - Functions, types and variables in the public test API should have the + ``tst_`` prefix. With some exceptions for symbols already prefixed with + ``safe_`` or ``ltp_``. + + Static (private) symbols should not have the prefix. + + * - LTP-004 + - Test executable symbols are marked ``static`` + - Test executables should not export symbols unnecessarily. This means + that all top-level variables and functions should be marked with the + ``static`` keyword. The only visible symbols should be those included + from shared object files. + + * - LTP-005 + - Array must terminate with a sentinel value (i.e. ``NULL`` or ``{}``) + - When defining arrays in the ``struct tst_test`` structure, we need to + end the array items with a sentinel ``NULL`` value. + +Shell coding style +------------------ + +When writing testcases in shell, write in *portable shell* only, it's a good +idea to try to run the test using alternative shell (alternative to bash, for +example dash) too. + +*Portable shell* means Shell Command Language as defined by POSIX with an +exception of few widely used extensions, namely **local** keyword used inside of +functions and ``-o`` and ``-a`` test parameters (that are marked as obsolete in +POSIX). + +You can either try to run the testcases in Debian which has ``/bin/sh`` pointing +to ``dash`` by default, or to install ``dash`` on your favorite distribution, +then use it to run tests. If your distribution lacks ``dash`` package, you can +always compile it from `sources <http://gondor.apana.org.au/~herbert/dash/files/>`_. + +Run ``make check`` in the test's directory and/or use ``make check-$TCID.sh``. +It uses (among other checks) our vendored version of +`checkbashism.pl <https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl>`_ +from Debian that is used to check for non-portable shell code. + +.. note:: + + If ``make check`` does not report any problems the code still may be wrong, + as ``checkbashisms.pl`` is used for checking only common mistakes. + +Here there are some common sense style rules for shell + +* Keep lines under 80 chars +* Use tabs for indentation +* Keep things simple, avoid unnecessary subshells +* Don't do confusing things (i.e. don't name your functions like common shell + commands, etc.) +* Quote variables +* Be consistent + +3 Backwards compatibility +~~~~~~~~~~~~~~~~~~~~~~~~~ + +LTP test should be as backward compatible as possible. Think of an enterprise +distributions with long term support (more than five years since the initial +release) or of an embedded platform that needs to use several years old +toolchain supplied by the manufacturer. + +Therefore LTP test for more current features should be able to cope with older +systems. It should at least compile fine and if it's not appropriate for the +configuration it should return ``TCONF``. + +There are several types of checks we use: + +* The *configure script* is usually used to detect availability of a function + declarations in system headers. It's used to disable tests at compile time or + to enable fallback definitions. + +* Checking the ``errno`` value is another type of runtime check. Most of the + syscalls returns either ``EINVAL`` or ``ENOSYS`` when syscall was not + implemented or was disabled upon kernel compilation. + +* LTP has kernel version detection that can be used to disable tests at runtime. + Unfortunately, the kernel version does not always corresponds to a well + defined feature set, as distributions tend to backport hundreds of patches + while the kernel version stays the same. Use with caution. + +* Lately, we added a kernel ``.config`` parser. A test can define a boolean + expression of kernel config variables that has to be satisfied in order to run + a test. At the moment, this is mostly used for kernel namespaces. + +* Sometimes it makes sense to define a few macros instead of creating a + configure test. One example is Linux specific POSIX clock ids in + ``include/lapi/posix_clocks.h`` + +Dealing with messed up legacy code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +LTP still contains a lot of old and messy code and we are cleaning it up as +fast as we can but, despite the decade of efforts, there is still a lot of it. +If you start modifying old or a messy testcase and your changes are more +complicated than simple typo fixes, you should convert the test into a new +library first. + +It's also much easier to review the changes if you split them into a smaller +logical groups. The same goes for moving files: if you need to rename or to move +files, do it in a separate patch. + +License +~~~~~~~ + +Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or +any later version). + +Use ``SPDX-License-Identifier: GPL-2.0-or-later`` + +LTP Structure +------------- + +The structure of LTP is quite simple. Each test is a binary written either in +portable shell or C. The test gets a configuration via environment variables +and/or command line parameters, it prints additional information into the +stdout and reports overall success/failure via the exit value. + +Tests are generally placed under the ``testcases/`` directory. Everything that +is a syscall or (slightly confusingly) libc syscall wrapper, goes under +``testcases/kernel/syscalls/``. + +There is also ``testcases/open_posix_testsuite/`` which is a well maintained +fork of the Open POSIX testsuite project, that has been dead since 2005. + +We also have a number of directories with tests for more specific features, such +as containers, etc. + +Runtest Files +~~~~~~~~~~~~~ + +The list of tests to be executed is stored in runtest files under the +``runtest/`` directory. The default set of runtest files to be executed is +stored in ``scenario_groups/default``. When you add a test, you should add +corresponding entries into some runtest file(s) as well. + +Each line of runtest file contains one test. The first item is the test name. +All other items, separated by space will be executed as a command. + +.. code-block:: bash + + shell_test01 echo "SUCCESS" | shell_pipe01.sh + splice02 splice02 -s 20 + +Blank lines and lines starting with a ``#`` (comments) are ignored. + +Syscalls tests, placed under ``testcases/kernel/syscalls/``, use +``runtest/syscalls`` file. For kernel related tests for memory management we +have ``runtest/mm``, etc. + +.. note:: + + runtest files should have one entry per a test. Creating a + wrapper that runs all your tests and adding it as a single test + into runtest file is strongly discouraged. + +Datafiles +--------- + +If your test needs data files, these should be put into a subdirectory +named ``datafiles`` and installed into the ``testcases/data/$TCID`` directory. +This will require to add ``INSTALL_DIR := testcases/data/TCID`` into +``datafiles/Makefile``. + +You can obtain path to datafiles via ``$TST_DATAROOT`` provided by ``test.sh`` +or via C function ``tst_dataroot()`` provided by libltp: + +.. code-block:: c + + const char *dataroot = tst_dataroot(); + +Datafiles can also be accessed as ``$LTPROOT/testcases/data/$TCID/...``, +but ``$TST_DATAROOT`` and ``tst_dataroot()`` are preferred, as these can be used +when running testcases directly in git tree as well as from install location. + +Subexecutables +~~~~~~~~~~~~~~ + +If your test needs to execute a binary, place it in the same directory of the +testcase and name the binary with ``$TESTNAME_`` prefix, where ``$TESTNAME`` is +the name of the test binary. Once the test is executed by the framework, the +path to the directory with all LTP binaries is added to the ``$PATH`` and you +can execute it via its name. + +.. note:: + + If you need to execute a test from the LTP tree, you can add ``PATH`` to + the current directory with ``PATH="$PATH:$PWD" ./foo01``. + +Test Contribution Checklist +--------------------------- + +#. Test compiles and it runs fine (check with ``-i 10`` and ``-i 0`` too) +#. ``make check`` should not emit any warnings for the test you are working on + (hint: run it in the test's directory and/or use ``make check-$TCID``) +#. The runtest entries are in place +#. New test binaries are added into the corresponding ``.gitignore`` files +#. Patches apply over the latest git + +About .gitignore files +~~~~~~~~~~~~~~~~~~~~~~ + +There are numerous ``.gitignore`` files in the LTP tree. Usually, there is a +``.gitignore`` file for a group of tests. The reason of this setup is simple: +it's easier to maintain a ``.gitignore`` file per tests' directory, rather +than having a single file in the project root directory. In this way, we don't +have to update all the gitignore files when moving directories, and they get +deleted automatically when a directory with tests is removed. + +Testing pre-release kernel features +----------------------------------- + +Tests for features not yet in the mainline kernel release are accepted. However, +they must be added only to the **staging** runtest file. Once a feature is part +of the stable kernel ABI, the associated test must be moved out of staging. + +Testing builds with GitHub Actions +---------------------------------- + +Master branch is tested in `GitHub Actions <https://github.com/linux-test-project/ltp/actions>`_ +to ensure LTP builds in various distributions, including old, current and +bleeding edge. ``gcc`` and ``clang`` toolchains are also tested for various +architectures using cross-compilation. For a full list of tested distros, please +check ``.github/workflows/ci.yml``. + +.. note:: + + Passing the GitHub Actions CI means that LTP compiles in a variety of + different distributions on their **newest releases**. + The CI also checks for code linting, running ``make check`` in the whole + LTP project. + +LTP C And Shell Test API Comparison +----------------------------------- + +.. list-table:: + :header-rows: 1 + + * - C API ``struct tst_test`` members + - Shell API ``$TST_*`` variables + + * - .all_filesystems + - TST_ALL_FILESYSTEMS + + * - .bufs + - \- + + * - .caps + - \- + + * - .child_needs_reinit + - not applicable + + * - .cleanup + - TST_CLEANUP + + * - .dev_extra_opts + - TST_DEV_EXTRA_OPTS + + * - .dev_fs_opts + - TST_DEV_FS_OPTS + + * - .dev_fs_type + - TST_FS_TYPE + + * - .dev_min_size + - not applicable + + * - .format_device + - TST_FORMAT_DEVICE + + * - .max_runtime + - \- + + * - .min_cpus + - not applicable + + * - .min_kver + - TST_MIN_KVER + + * - .min_mem_avail + - not applicable + + * - .mnt_flags + - TST_MNT_PARAMS + + * - .min_swap_avail + - not applicable + + * - .mntpoint | .mnt_data + - TST_MNTPOINT + + * - .mount_device + - TST_MOUNT_DEVICE + + * - .needs_cgroup_ctrls + - \- + + * - .needs_checkpoints + - TST_NEEDS_CHECKPOINTS + + * - .needs_cmds + - TST_NEEDS_CMDS + + * - .needs_devfs + - \- + + * - .needs_device + - TST_NEEDS_DEVICE + + * - .needs_drivers + - TST_NEEDS_DRIVERS + + * - .needs_kconfigs + - TST_NEEDS_KCONFIGS + + * - .needs_overlay + - \- + + * - .needs_rofs + - \- + + * - .needs_root + - TST_NEEDS_ROOT + + * - .needs_tmpdir + - TST_NEEDS_TMPDIR + + * - .options + - TST_PARSE_ARGS | TST_OPTS + + * - .resource_files + - \- + + * - .restore_wallclock + - not applicable + + * - .sample + - \- + + * - .save_restore + - \- + + * - .scall + - not applicable + + * - .setup + - TST_SETUP + + * - .skip_filesystems + - TST_SKIP_FILESYSTEMS + + * - .skip_in_compat + - \- + + * - .skip_in_lockdown + - TST_SKIP_IN_LOCKDOWN + + * - .skip_in_secureboot + - TST_SKIP_IN_SECUREBOOT + + * - .supported_archs + - not applicable + + * - .tags + - \- + + * - .taint_check + - \- + + * - .tcnt + - TST_CNT + + * - .tconf_msg + - not applicable + + * - .test | .test_all + - TST_TESTFUNC + + * - .test_variants + - \- + + * - .timeout + - TST_TIMEOUT + + * - .tst_hugepage + - not applicable + + * - .ulimit + - not applicable + + * - not applicable + - TST_NEEDS_KCONFIGS_IFS + + * - not applicable + - TST_NEEDS_MODULE + + * - not applicable + - TST_POS_ARGS + + * - not applicable + - TST_USAGE + +.. list-table:: + :header-rows: 1 + + * - C API other structs + - Shell API ``$TST_*`` variables + + * - struct tst_device + - TST_DEVICE diff --git a/doc/index.rst b/doc/index.rst new file mode 100644 index 000000000..80820ab51 --- /dev/null +++ b/doc/index.rst @@ -0,0 +1,103 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +.. include:: ../README.rst + +.. toctree:: + :maxdepth: 3 + :hidden: + :caption: For users + + users/quick_start + users/setup_tests + users/supported_systems + users/stats + +.. toctree:: + :maxdepth: 3 + :hidden: + :caption: For developers + + developers/setup_mailinglist + developers/writing_tests + developers/test_case_tutorial + developers/api_c_tests + developers/api_shell_tests + developers/api_network_tests + developers/api_kvm_tests + developers/ltp_library + developers/build_system + developers/debugging + +.. toctree:: + :maxdepth: 3 + :hidden: + :caption: For maintainers + + maintainers/patch_review + maintainers/ltp_release_procedure + +For users +--------- + +.. descriptions here are active + +:doc:`users/quick_start` + How to build and use LTP framework in few steps + +:doc:`users/setup_tests` + How to setup tests execution + +:doc:`users/supported_systems` + A list of supported technologies by the LTP framework + +:doc:`users/stats` + Some LTP statistics + +For developers +-------------- + +.. descriptions here are active + +:doc:`developers/setup_mailinglist` + How to configure git and to start sending patches via ``git send-email`` + +:doc:`developers/writing_tests` + Starting guide on writing tests + +:doc:`developers/test_case_tutorial` + A tutorial showing how to write a test from scratch using C API + +:doc:`developers/api_c_tests` + Walk through the C API features + +:doc:`developers/api_shell_tests` + Walk through the Shell API features + +:doc:`developers/api_network_tests` + Walk through the network API features + +:doc:`developers/api_kvm_tests` + Walk through the KVM API features + +:doc:`developers/ltp_library` + Developing new features in the LTP library + +:doc:`developers/build_system` + Short introduction to the LTP build system + +:doc:`developers/debugging` + How to debug LTP tests + +For maintainers +--------------- + +:doc:`maintainers/patch_review` + Steps to follow when reviewing patches + +:doc:`maintainers/ltp_release_procedure` + Steps to follow for a new LTP release + + +Getting help +------------ +To report a problem or suggest any feature, please write at ltp@lists.linux.it diff --git a/doc/maintainers/ltp_release_procedure.rst b/doc/maintainers/ltp_release_procedure.rst new file mode 100644 index 000000000..f0bc01cee --- /dev/null +++ b/doc/maintainers/ltp_release_procedure.rst @@ -0,0 +1,173 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Release process +=============== + +Preparations +------------ + +The release procedure generally takes a few weeks. In the first week or two, +patches that should go into the release are reviewed and possibly merged. These +patches are either fixes or patches pointed out by the community. + +Patch review, when finished, is followed by a git freeze, which is a period +where only fixes are pushed to the git. During that period community is +expected to run a LTP pre-release tests, reports problems, and/or send fixes to +the mailing list. In this period we are especially making sure that there are +no regressions in the test results on a wide range of distributions and +architectures. + +Once the stabilization period has ended the time has finally come to proceed +with the release. + +Prepare the release notes +------------------------- + +Part of the preparation is also to write the release notes, which are then +added to the GitHub release and also sent as announcement to various mailing +lists (see below). + +Have a look at `this release letter <https://lore.kernel.org/ltp/ZGNiQ1sMGvPU_ETp@yuki/>`_ +to get the idea how it should look. + +Tag the git and push changes to github +-------------------------------------- + +.. code-block:: bash + + cd ltp + echo YYYYMMDD > VERSION + git commit -S -s -m 'LTP YYYYMMDD' VERSION + git tag -s -a YYYYMMDD -m 'LTP YYYYMMDD' + git push origin master:master + git push origin YYYYMMDD + +The string ``YYYYMMDD`` should be substituted to the current date. + +You can use `tools/tag-release.sh <https://github.com/linux-test-project/ltp/blob/master/tools/tag-release.sh>`_ +script to have the above automated process. +It allows you to verify the tag before pushing it and does other checks. + +.. code-block:: bash + + $ ./tools/tag-release.sh + ===== git push ===== + new tag: 'YYYYMMDD', previous tag: '20230127' + tag YYYYMMDD + Tagger: Person-who-released LTP <foo@example.com> + Date: ... + + LTP YYYYMMDD + -----BEGIN PGP SIGNATURE----- + ... + -----END PGP SIGNATURE----- + + commit 3ebc2dfa85c2445bb68d8c0d66e33c4da1e1b3a7 + gpg: using RSA key ... + ... + Primary key fingerprint: ... + Author: Person-who-released LTP <foo@example.com> + Date: ... + + LTP YYYYMMDD + + Signed-off-by: Person-who-released LTP <foo@example.com> + + diff --git a/VERSION b/VERSION + index af4c41fec..ae488c0e7 100644 + --- a/VERSION + +++ b/VERSION + @@ -1 +1 @@ + -20230127 + +YYYYMMDD + + Please check tag and signature. Proceed? [N/y]: y + Pushing changes to upstream git. Proceed? [N/y]: y + ... + To github.com:linux-test-project/ltp.git + * [new tag] YYYYMMDD -> YYYYMMDD + +Prepare tarballs and metadata documentation +------------------------------------------- + +The following procedure will show how to create the release archives and the +metadata documentation: + +.. code-block:: bash + + # clone already clonned git repository to new folder + cd .. + git clone ltp ltp-full-YYYYMMDD + cd ltp-full-YYYYMMDD + + # update all submodules + git submodule update --init + + # Generate configure script + make autotools + + # Generate tarballs + cd .. + tar -cjf ltp-full-YYYYMMDD.tar.bz2 ltp-full-YYYYMMDD --exclude .git + tar -cJf ltp-full-YYYYMMDD.tar.xz ltp-full-YYYYMMDD --exclude .git + + # Generate checksums + md5 ltp-full-YYYYMMDD.tar.xz > ltp-full-YYYYMMDD.tar.xz.md5 + sha1 ltp-full-YYYYMMDD.tar.xz > ltp-full-YYYYMMDD.tar.xz.sha1 + sha256sum ltp-full-YYYYMMDD.tar.xz > ltp-full-YYYYMMDD.tar.xz.sha256 + + # Generate metadata documentation + ./configure --with-metadata-generator=asciidoctor + make -C metadata + cp -v docparse/metadata.html ../metadata.YYYYMMDD.html + +You can use `tools/create-tarballs-metadata.sh <https://github.com/linux-test-project/ltp/blob/master/tools/create-tarballs-metadata.sh>`_ +script to have the above procedure automated. All generated files are placed +in the ``ltp-release-YYYYMMDD`` directory. + +.. code-block:: bash + + $ ./tools/create-tarballs-metadata.sh + ===== git clone ===== + Cloning into 'ltp-full-YYYYMMDD'... + done. + ===== Update submodules ===== + Submodule 'tools/kirk' (https://github.com/linux-test-project/kirk.git) registered for path 'tools/kirk' + ... + ===== Generate configure script ===== + sed -n '1{s:LTP-:m4_define([LTP_VERSION],[:;s:$:]):;p;q}' VERSION > m4/ltp-version.m4 + aclocal -I m4 + ... + ===== Generate tarballs ===== + ===== Generate checksums ===== + ===== Generate metadata documentation ===== + checking for a BSD-compatible install... /usr/bin/install -c + ... + 'docparse/metadata.html' -> '/home/foo/ltp-release-YYYYMMDD/metadata.YYYYMMDD.html' + Generated files are in '/home/foo/ltp-release-YYYYMMDD', upload them to github + +Upload the generated files to GitHub +------------------------------------ + +Click on `Releases <https://github.com/linux-test-project/ltp/releases>`_, then +switch to `Tags <https://github.com/linux-test-project/ltp/tags>`_. Click on +'Add release notes'. There should be 'Attach binaries ...' link at the +bottom of the page. + +Don't forget to upload checksums for the tarballs and metadata documentation +as well. + +Send release announcement +------------------------- + +The announcement is sent to: + +* ltp at lists.linux.it +* linux-kernel at vger.kernel.org +* libc-alpha at sourceware.org + +CCed to: + +* lwn at lwn.net +* akpm at linux-foundation.org +* torvalds at linux-foundation.org diff --git a/doc/maintainers/patch_review.rst b/doc/maintainers/patch_review.rst new file mode 100644 index 000000000..81d198532 --- /dev/null +++ b/doc/maintainers/patch_review.rst @@ -0,0 +1,171 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Patch review +============ + +Anyone can and should review patches. It's the only way to get good at patch +review and for the project to scale. For this reason, we have a short guide on +what to do during the review process. + +Goals of patch review +--------------------- + +#. Prevent false positive test results +#. Prevent false negative test results +#. Keep the code as simple as possible, but no simpler + +How to find clear errors +------------------------ + +A clear error is one where there is unlikely to be any argument if you +provide evidence of it. Evidence being an error trace or logical proof +that an error will occur in a common situation. + +The following are examples and may not be appropriate for all tests. + +* Merge the patch locally. It should apply cleanly to master. +* Compile the patch with default and non-default configurations. + + * Use sanitizers e.g. undefined behaviour, address. + * Compile on non-x86 + * Compile on x86 with ``-m32`` + * Compile testing patches with GitHub Actions in LTP repo fork can cover + various distros/architectures + +* Use ``make check`` +* Run effected tests in a VM + + * Use single vCPU + * Use many vCPUs and enable NUMA + * Restrict RAM to < 1GB. + +* Run effected tests on an embedded device +* Run effected tests on non-x86 machine in general +* Run reproducers on a kernel where the bug is present +* Run tests with ``-i0`` and ``-i2`` +* Compare usage of system calls with man page descriptions +* Compare usage of system calls with kernel code +* Double check commit message +* Search the LTP library for existing helper functions +* Check doc formatting, i.e. ``make doc && chromium docparse/metadata.html`` + +How to find subtle errors +------------------------- + +A subtle error is one where you can expect some argument because you +do not have clear evidence of an error. It is best to state these as +questions and not make assertions if possible. + +Although if it is a matter of style or "taste" then senior maintainers +can assert what is correct to avoid bike shedding. + +* Ask what happens if there is an error, could it be debugged just + with the test output? +* Are we testing undefined behavior? + + * Could future kernel behaviour change without "breaking userland"? + * Does the kernel behave differently depending on hardware? + * Does it behave differently depending on kernel configuration? + * Does it behave differently depending on the compiler? + * Would it behave differently if the order of checks on syscall parameters + changed in the kernel? + +* Will it scale to tiny and huge systems? + + * What happens if there are 100+ CPUs? + * What happens if each CPU core is very slow? + * What happens if there are 2TB of RAM? + +* Are we repeating a pattern that can be turned into a library function? +* Is a single test trying to do too much? +* Could multiple similar tests be merged? +* Race conditions + + * What happens if a process gets preempted? + * Could checkpoints or fuzzsync by used instead? + * Note, usually you can insert a sleep to prove a race condition + exists however finding them is hard + +* Is there a simpler way to achieve the same kernel coverage? + +How to get patches merged +------------------------- + +Once you think a patch is good enough you should add your ``Reviewed-by`` +and/or ``Tested-by`` tags. This means you will get some credit for getting +the patch merged. Also some blame if there are problems. + +If you ran the test you can add the ``Tested-by`` tag. If you read the +code or used static analysis tools on it, you can add the Reviewed-by +tag. + +In addition you can expect others to review your patches and add their +tags. This will speed up the process of getting your patches merged. + +Maintainers Checklist +--------------------- + +Patchset should be tested locally and ideally also in maintainer's fork in +GitHub Actions on GitHub. + +.. note:: + + GitHub Actions do only build testing, passing the CI means only that + the test compiles fine on variety of different distributions and releases. + +The test should be executed at least once locally and should PASS as well. + +Commit messages should have + +* Author's ``Signed-off-by`` tag +* Committer's ``Reviewed-by`` or ``Signed-off-by`` tag +* Check also mailing lists for other reviewers / testers tags, notes and failure + reports +* ``Fixes: hash`` if it fixes particular LTP commit +* ``Fixes: #N`` if it fixes github issue number N, so it's automatically closed +* LTP documentation should be kept up to date. + +After patch is accepted or rejected, set correct state and archive in the +`LTP patchwork instance <https://patchwork.ozlabs.org/project/ltp/list/>`_. + +New tests +--------- + +New test should + +* Have a record in runtest file +* Test should work fine with more than one iteration (e.g. run with ``-i 100``) +* Run with ``-i 0`` to check that setup and cleanup are coded properly + (no test is being run) +* Have a brief description +* License: the default license for new tests is GPL v2 or later, use + ``GPL-2.0-or-later``; the licence for test (e.g. GPL-2.0) should not change + unless test is completely rewritten +* Old copyrights should be kept unless test is completely rewritten + +C tests +~~~~~~~ + +* Use the new C API, implementing ``struct tst_test`` +* Test binaries are added into corresponding ``.gitignore`` files +* Check coding style with ``make check`` +* Docparse documentation +* If a test is a regression test it should include ``.tags`` in the + ``struct tst_test`` definition +* When rewriting old tests, `uClinux <https://en.wikipedia.org/wiki/%CE%9CClinux>`_ + support should be removed (project has been discontinued). + E.g. remove ``#ifdef UCLINUX``, replace ``FORK_OR_VFORK()`` with simple + ``fork()`` or ``SAFE_FORK()``. + +Shell tests +~~~~~~~~~~~ + +* Use new shell API +* Check coding style with ``make check`` +* If a test is a regression test it should include related kernel or glibc + commits as a comment + +LTP library +~~~~~~~~~~~ + +For patchset touching the LTP library, follow the LTP library guidelines. diff --git a/doc/Build-System.asciidoc b/doc/old/Build-System.asciidoc similarity index 100% rename from doc/Build-System.asciidoc rename to doc/old/Build-System.asciidoc diff --git a/doc/C-Test-API.asciidoc b/doc/old/C-Test-API.asciidoc similarity index 100% rename from doc/C-Test-API.asciidoc rename to doc/old/C-Test-API.asciidoc diff --git a/doc/C-Test-Case-Tutorial.asciidoc b/doc/old/C-Test-Case-Tutorial.asciidoc similarity index 100% rename from doc/C-Test-Case-Tutorial.asciidoc rename to doc/old/C-Test-Case-Tutorial.asciidoc diff --git a/doc/C-Test-Network-API.asciidoc b/doc/old/C-Test-Network-API.asciidoc similarity index 100% rename from doc/C-Test-Network-API.asciidoc rename to doc/old/C-Test-Network-API.asciidoc diff --git a/doc/Contact-Info.asciidoc b/doc/old/Contact-Info.asciidoc similarity index 100% rename from doc/Contact-Info.asciidoc rename to doc/old/Contact-Info.asciidoc diff --git a/doc/KVM-Test-API.asciidoc b/doc/old/KVM-Test-API.asciidoc similarity index 100% rename from doc/KVM-Test-API.asciidoc rename to doc/old/KVM-Test-API.asciidoc diff --git a/doc/LTP-Library-API-Writing-Guidelines.asciidoc b/doc/old/LTP-Library-API-Writing-Guidelines.asciidoc similarity index 100% rename from doc/LTP-Library-API-Writing-Guidelines.asciidoc rename to doc/old/LTP-Library-API-Writing-Guidelines.asciidoc diff --git a/doc/LTP-Release-Procedure.asciidoc b/doc/old/LTP-Release-Procedure.asciidoc similarity index 100% rename from doc/LTP-Release-Procedure.asciidoc rename to doc/old/LTP-Release-Procedure.asciidoc diff --git a/doc/Maintainer-Patch-Review-Checklist.asciidoc b/doc/old/Maintainer-Patch-Review-Checklist.asciidoc similarity index 100% rename from doc/Maintainer-Patch-Review-Checklist.asciidoc rename to doc/old/Maintainer-Patch-Review-Checklist.asciidoc diff --git a/doc/old/Makefile b/doc/old/Makefile new file mode 100644 index 000000000..f7e4dd021 --- /dev/null +++ b/doc/old/Makefile @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (C) 2009, Cisco Systems Inc. +# Ngie Cooper, July 2009 + +top_srcdir ?= .. + +include $(top_srcdir)/include/mk/env_pre.mk + +RECURSIVE_TARGETS := install + +include $(top_srcdir)/include/mk/generic_trunk_target.mk diff --git a/doc/Shell-Test-API.asciidoc b/doc/old/Shell-Test-API.asciidoc similarity index 100% rename from doc/Shell-Test-API.asciidoc rename to doc/old/Shell-Test-API.asciidoc diff --git a/doc/Supported-kernel,-libc,-toolchain-versions.asciidoc b/doc/old/Supported-kernel,-libc,-toolchain-versions.asciidoc similarity index 100% rename from doc/Supported-kernel,-libc,-toolchain-versions.asciidoc rename to doc/old/Supported-kernel,-libc,-toolchain-versions.asciidoc diff --git a/doc/Test-Writing-Guidelines.asciidoc b/doc/old/Test-Writing-Guidelines.asciidoc similarity index 100% rename from doc/Test-Writing-Guidelines.asciidoc rename to doc/old/Test-Writing-Guidelines.asciidoc diff --git a/doc/User-Guidelines.asciidoc b/doc/old/User-Guidelines.asciidoc similarity index 100% rename from doc/User-Guidelines.asciidoc rename to doc/old/User-Guidelines.asciidoc diff --git a/doc/man1/Makefile b/doc/old/man1/Makefile similarity index 100% rename from doc/man1/Makefile rename to doc/old/man1/Makefile diff --git a/doc/man1/doio.1 b/doc/old/man1/doio.1 similarity index 100% rename from doc/man1/doio.1 rename to doc/old/man1/doio.1 diff --git a/doc/man1/iogen.1 b/doc/old/man1/iogen.1 similarity index 100% rename from doc/man1/iogen.1 rename to doc/old/man1/iogen.1 diff --git a/doc/man1/ltp-bump.1 b/doc/old/man1/ltp-bump.1 similarity index 100% rename from doc/man1/ltp-bump.1 rename to doc/old/man1/ltp-bump.1 diff --git a/doc/man1/ltp-pan.1 b/doc/old/man1/ltp-pan.1 similarity index 100% rename from doc/man1/ltp-pan.1 rename to doc/old/man1/ltp-pan.1 diff --git a/doc/man3/Makefile b/doc/old/man3/Makefile similarity index 100% rename from doc/man3/Makefile rename to doc/old/man3/Makefile diff --git a/doc/man3/parse_opts.3 b/doc/old/man3/parse_opts.3 similarity index 100% rename from doc/man3/parse_opts.3 rename to doc/old/man3/parse_opts.3 diff --git a/doc/man3/parse_ranges.3 b/doc/old/man3/parse_ranges.3 similarity index 100% rename from doc/man3/parse_ranges.3 rename to doc/old/man3/parse_ranges.3 diff --git a/doc/man3/random_range.3 b/doc/old/man3/random_range.3 similarity index 100% rename from doc/man3/random_range.3 rename to doc/old/man3/random_range.3 diff --git a/doc/man3/random_range_seed.3 b/doc/old/man3/random_range_seed.3 similarity index 100% rename from doc/man3/random_range_seed.3 rename to doc/old/man3/random_range_seed.3 diff --git a/doc/man3/tst_res.3 b/doc/old/man3/tst_res.3 similarity index 100% rename from doc/man3/tst_res.3 rename to doc/old/man3/tst_res.3 diff --git a/doc/man3/tst_sig.3 b/doc/old/man3/tst_sig.3 similarity index 100% rename from doc/man3/tst_sig.3 rename to doc/old/man3/tst_sig.3 diff --git a/doc/man3/tst_tmpdir.3 b/doc/old/man3/tst_tmpdir.3 similarity index 100% rename from doc/man3/tst_tmpdir.3 rename to doc/old/man3/tst_tmpdir.3 diff --git a/doc/man3/usctest.3 b/doc/old/man3/usctest.3 similarity index 100% rename from doc/man3/usctest.3 rename to doc/old/man3/usctest.3 diff --git a/doc/namespaces-helper-tools.txt b/doc/old/namespaces-helper-tools.txt similarity index 100% rename from doc/namespaces-helper-tools.txt rename to doc/old/namespaces-helper-tools.txt diff --git a/doc/nommu-notes.txt b/doc/old/nommu-notes.txt similarity index 100% rename from doc/nommu-notes.txt rename to doc/old/nommu-notes.txt diff --git a/doc/rules.tsv b/doc/old/rules.tsv similarity index 100% rename from doc/rules.tsv rename to doc/old/rules.tsv diff --git a/doc/requirements.txt b/doc/requirements.txt new file mode 100644 index 000000000..742fb8b4b --- /dev/null +++ b/doc/requirements.txt @@ -0,0 +1,3 @@ +sphinx-rtd-theme==2.0.0 +linuxdoc==20231020 +sphinxcontrib-spelling==7.7.0 diff --git a/doc/spelling_wordlist b/doc/spelling_wordlist new file mode 100644 index 000000000..0c016e615 --- /dev/null +++ b/doc/spelling_wordlist @@ -0,0 +1,120 @@ +aarch +akpm +amd +archiver +archs +args +asciidoc +autoconf +automake +backport +baz +btime +bufs +CCed +cgroup +cgroups +checksum +checksums +cmds +codebase +compat +config +cpus +ctrls +datafiles +dev +devfs +dio +distro +distros +docparse +doio +executables +fd +filesystem +filesystems +fixup +fs +fstrict +fuzzsync +gcc +gdb +github +gitignore +glibc +growfiles +hugepage +iogen +kbuild +kconfigs +kver +ld +le +libc +libltp +libs +linkable +linux +lockdown +ltp +lwn +Maipo +Makefile +Makefiles +mem +mnt +mntpoint +msg +namespace +namespaces +onwards +openSUSE +patchset +pkgconf +posix +ppc +pre +pre +preprocessor +rebase +reflog +reinit +repo +rofs +runtest +scall +secureboot +sourceware +statx +stdout +structs +stx +subdirectory +subshell +subshells +syscall +syscalls +tcnt +tconf +teardown +testcase +testcases +testsuite +testsuites +tmp +tmpdir +toolchain +toolchains +torvalds +tst +uClibc +ulimit +un +userland +userspace +vCPU +vCPUs +vendoring +vger +wallclock diff --git a/doc/users/quick_start.rst b/doc/users/quick_start.rst new file mode 100644 index 000000000..40d8dd9c6 --- /dev/null +++ b/doc/users/quick_start.rst @@ -0,0 +1,126 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Installation and tests execution +================================ + +Basics requirements to build LTP are the following: + +* git +* autoconf +* automake +* make +* gcc +* m4 +* pkgconf / pkg-config +* libc headers +* linux headers + +.. code-block:: console + + $ git clone https://github.com/linux-test-project/ltp.git + $ cd ltp + $ make autotools + $ ./configure + +.. note:: + + For optional library dependencies, take a look at the scripts inside ``ci/`` + directory. + +Running single tests +-------------------- + +LTP provides the possibility to build and run single tests: + +.. code-block:: console + + $ cd testcases/kernel/syscalls/foo + $ make + $ PATH=$PATH:$PWD ./foo01 + +Shell testcases are a bit more complicated, since they need to setup ``PATH`` +as well as to compiled binary helpers: + +.. code-block:: console + + $ cd testcases/lib + $ make + $ cd ../commands/foo + $ PATH=$PATH:$PWD:$PWD/../../lib/ ./foo01.sh + +Open Posix Testsuite has it's own build system which needs Makefiles to be +generated first: + +.. code-block:: console + + $ cd testcases/open_posix_testsuite/ + $ make generate-makefiles + $ cd conformance/interfaces/foo + $ make + $ ./foo_1-1.run-test + +Compiling and installing all testcases +-------------------------------------- + +To compile all tests is really simple: + +.. code-block:: console + + $ make + + $ # install LTP inside /opt/ltp by default + $ make install + +.. note:: + + Some tests will be disabled if ``configure`` script won't find the build + dependencies. + +Running tests +------------- + +To run all the test suites + +.. code-block:: console + + $ cd /opt/ltp + + $ # run syscalls testing suite + $ ./kirk -f ltp -r syscalls + +.. note:: + + Many test cases have to be executed as root. + +Test suites (e.g. syscalls) are defined in the ``runtest`` directory. Each file +contains a list of test cases in a simple format. + +Each test case has its own executable or script that can directly executed: + +.. code-block:: console + + $ testcases/bin/abort01 + + $ # some tests have arguments + $ testcases/bin/mesgq_nstest -m none + + $ # vast majority of tests have a help + $ testcases/bin/ioctl01 -h + + $ # Many require certain environment variables to be set + $ LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh + +Most commonly, the ``PATH`` variable needs to be set and also ``LTPROOT``, but +there are a number of other variables which usually ``kirk`` sets for you. + +.. note:: + + All shell scripts need the ``PATH`` to be set. However, this is not limited + to shell scripts and some C based tests need environment variables as well. + They usually raise a configuration error when this is needed. + +Network tests +------------- + +Network tests usually require a certain setup that is described in +``testcases/network/README.md``. diff --git a/doc/users/setup_tests.rst b/doc/users/setup_tests.rst new file mode 100644 index 000000000..910cc2315 --- /dev/null +++ b/doc/users/setup_tests.rst @@ -0,0 +1,104 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Tests setup +=========== + +The internal LTP library provides a set of features that permits to customize +tests behaviour by setting environment variables and using specific tests +arguments. + +Library environment variables +----------------------------- + +Following environment variables are expected to be set by LTP users. Therefore, +with some exceptions, they have ``LTP_`` prefix. Environment variables with +``TST_`` prefix are used inside LTP shell API and should **not** be set by +users. + +.. list-table:: + :header-rows: 1 + + * - Variable + - Note + + * - KCONFIG_PATH + - The path to the kernel config file, (if not set, it tries the usual paths + ``/boot/config-RELEASE`` or ``/proc/config.gz``) + + * - KCONFIG_SKIP_CHECK + - Skip kernel config check if variable set (not set by default) + + * - LTPROOT + - Prefix for installed LTP. **Should be always set**, since some tests + need it to use data files (``LTP_DATAROOT``). LTP is by default installed + into ``/opt/ltp`` + + * - LTP_COLORIZE_OUTPUT + - By default LTP colorizes it's output unless it's redirected to a pipe or + file. Force colorized output behaviour: ``y`` or ``1``: always colorize, + ``n`` or ``0``: never colorize. + + * - LTP_DEV + - Path to the block device to be used. C Language: ``.needs_device = 1``. + Shell language: ``TST_NEEDS_DEVICE=1``. + + * - LTP_SINGLE_FS_TYPE + - Testing only - specifies filesystem instead all supported + (for tests with ``.all_filesystems``). + + * - LTP_DEV_FS_TYPE + - Filesystem used for testing (default: ``ext2``). + + * - LTP_TIMEOUT_MUL + - Multiplies timeout, must be number >= 0.1 (> 1 is useful for slow + machines to avoid unexpected timeout). + + * - LTP_RUNTIME_MUL + - Multiplies maximal test iteration runtime. Tests that run for more than a + second or two are capped on runtime. You can scale the default runtime + both up and down with this multiplier. This is not yet implemented in the + shell API. + + * - LTP_VIRT_OVERRIDE + - Overrides virtual machine detection in the test library. Setting it to + empty string, tella the library that system is not a virtual machine. + Other possible values are ``kvm``, ``xen``, ``zvm`` and ``microsoft`` + that describe different types supervisors. + + * - PATH + - It's required to addjust path: ``PATH="$PATH:$LTPROOT/testcases/bin"`` + + * - TMPDIR + - Base directory for template directory (C language: ``.needs_tmpdir = 1`` + and shell: ``TST_NEEDS_TMPDIR=1``). Must be an absolute path (default: + '/tmp'). + + * - LTP_NO_CLEANUP + - Disable running test cleanup (defined in ``TST_CLEANUP``). + Shell API only. + + * - LTP_ENABLE_DEBUG + - Enable debug info (value ``1`` or ``y``). + +Test execution time and timeout +------------------------------- + +The limit on how long a test can run does compose of two parts: ``max_runtime`` +and ``timeout``. The limit does apply to a single test variant. That means, for +example, that tests which run for all available filesystems will apply this +limit for a single filesystem only. + +The ``max_runtime`` is a cap on how long the ``run()`` function can take and for +most testcases this part is set to zero. For tests that do run for more than a +second or two the ``max_runtime`` has to be defined and the ``run()`` function +has to check actively how much runtime is left. + +Test runtime can be scaled up and down with ``LTP_RUNTIME_MUL`` environment +variable or set on a commandline by the ``-I`` parameter. Hoewever, +setting the runtime too low will cause long running tests to exit prematurely, +possibly before having a chance to actually test anything. + +The timeout is a limit for test setup and cleanup and it's also a safety +margin for the runtime accounting. It's currently set to 30 seconds but it may +change later. If your target machine is too slow, it can be scaled up with the +``LTP_TIMEOUT_MUL`` environment variable. diff --git a/doc/users/stats.rst b/doc/users/stats.rst new file mode 100644 index 000000000..7073442aa --- /dev/null +++ b/doc/users/stats.rst @@ -0,0 +1,9 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Statistics +========== + +In this section we collect some statistics related to the current state of +LTP tests. + +.. include:: ../_static/syscalls.rst diff --git a/doc/users/supported_systems.rst b/doc/users/supported_systems.rst new file mode 100644 index 000000000..6220a96e0 --- /dev/null +++ b/doc/users/supported_systems.rst @@ -0,0 +1,110 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Supported systems +================= + +Kernel version +-------------- + +Minimal supported kernel version is **3.10**. + +Oldest tested distributions +--------------------------- + +.. list-table:: + :header-rows: 1 + + * - Distro + - Kernel + - glibc + - gcc + - clang + + * - openSUSE Leap 42.2 + - 4.4 + - 2.22 + - 4.8.5 + - \- + + * - Ubuntu 18.04 LTS bionic + - 4.15 + - 2.27 + - 7.3.0 + - \- + + * - Debian 11 (bullseye) + - 5.10 + - 2.31 + - 10.2.1 + - 11.0.1 + + * - RHEL-7.9 Maipo + - 3.10 + - 4.8.5 + - 2.17 + - \- + +Older distributions are not officially supported, which means that it +may or may not work. It all depends on your luck. It should be possible +to compile latest LTP even on slightly older distributions than we +support with a few manual tweaks, e.g. disabling manually tests for +newly added syscalls, etc. **Trivial fixes/workarounds may be accepted, +but users are encouraged to move to a newer distro.** + +If latest LTP cannot be compiled even with some amount of workarounds, +you may result to older LTP releases, however these are **not** supported +in any way. Also if you are trying to run LTP on more than 10 years old +distribution you may as well reconsider you life choices. + +Tested architectures +-------------------- + +.. list-table:: + :header-rows: 1 + + * - Architecture + - Build + + * - x86_64 + - native + + * - x86 emulation + - native + + * - aarch64 + - cross compilation + + * - ppc64le + - cross compilation + + * - s390x + - cross compilation + +Supported C libraries +--------------------- + +.. list-table:: + :header-rows: 1 + + * - C library + - Note + + * - `glibc <https://www.gnu.org/software/libc/>`_ + - Targeted libc, tested both compilation and actual test results. + + * - `uClibc-ng <https://uclibc-ng.org/>`_ + - Although not being tested, it should work as it attempt to maintain a glibc compatible interface. + + * - `uClibc <https://www.uclibc.org/>`_ + - Older uClibc might have problems. + + * - `musl <https://musl.libc.org/>`_ + - Not yet fully supported. Check ``ci/alpine.sh`` script. + + * - Android + - Please use `AOSP fork <https://android.googlesource.com/platform/external/ltp>`_ + +C version +--------- + +LTP is compiled with ``-std=gnu99``.