| Message ID | 20230308231810.2143384-2-hauke@hauke-m.de |
|---|---|
| State | Under Review |
| Delegated to: | Hauke Mehrtens |
| Headers | show |
| Series | [uci,1/2] fuzz: Compile using libstd++ | expand |
Hauke Mehrtens <hauke@hauke-m.de> [2023-03-09 00:18:10]: Hi, thanks for taking care, LGTM for a start. I'll just provide my past experience, something to consider as we're likely going to bump into those in the long term, so ideally take them into the account in the long term. > clang 14 generates debug information in DWARF 5 format, but valgrind > 19.0 does not support that. Install valgrind 20.0 from experimental > which supports it. IMO we should aim for reproducible results, thus we should likely provide tools containers with a known versions inside, so anyone is able to get same results using the source code Git hash and tool container version. Your current approach is highly dynamic, so if you're lucky enough, you might get a green CI light in the PR branch as the pipeline was run for example 7 days ago, so you're going to merge it into the upstream branch, but then it's going to fail as meanwhile Debian has bumped several packages and you're going to get a CI pipeline failure. IMO there should be a tools container Git repository, where we could build, test and provide versioned containers, for example: ghcr.io/openwrt/ci-tools/native-testing/clang14:20230305 ghcr.io/openwrt/ci-tools/native-testing/clang15:20230305 ghcr.io/openwrt/ci-tools/native-testing/gcc-8:20230305 ghcr.io/openwrt/ci-tools/native-testing/gcc-11:20230305 ghcr.io/openwrt/ci-tools/native-testing/gcc-12:20230305 We want to test with with multiple compiler versions as those tested changes might be backported into stable branches, we should even consider to have stable branches in every project so we could CI test them there as well, folks would simply create a backport PR in the respective project. > +++ b/.github/workflows/test.yml > @@ -0,0 +1,83 @@ > +name: 'Run tests' We've like 30+ C projects which we should likely cover in the end, thus considering possible additional 2 stable branches in each, it's around 60+ of similar workflow files duplicated in various repositories. I would consider this future maintenance overhead (imagine just a simple change or a fix being propagated into 60+ repositories/branches), so I would recommend to create a shareable Github Action instead: uses: openwrt/gh-actions-ci-native env: CI_NATIVE_TOOLCHAIN: clang14 uses: openwrt/gh-actions-ci-sdk env: CI_TARGET_SDK_RELEASE: master CI_TARGET_SDK: ath79-generic CI_TARGET_BUILD_DEPENDS: uci You can take a look at the ucode project for a very dated, but still working GitHub example, some bits are even present in uci repsitory in the .gitlab-ci.yml file. > + - name: Checkout libubox > + uses: actions/checkout@v3 > + with: > + repository: openwrt/libubox > + path: libubox This looks like another source of unreliability, similar to the toolchain versions above. For the start, I would simply include all those dependencies in the native toolchain container, thus we would need to have separate containers for each branch: ghcr.io/openwrt/ci-tools/native-testing/clang14_snapshot:20230305 ghcr.io/openwrt/ci-tools/native-testing/clang14_openwrt-22.03:20230305 ghcr.io/openwrt/ci-tools/native-testing/clang14_openwrt-21.02:20230305 ghcr.io/openwrt/ci-tools/native-testing/distro_snapshot:20230305 (distro default gcc12) ghcr.io/openwrt/ci-tools/native-testing/distro_21.02:20230305 (gcc8) ghcr.io/openwrt/ci-tools/native-testing/distro_22.03:20230305 (gcc11) So in the end, ideally, interested developer can have the same CI failure/result locally and debug/fix it faster: $ git clone https://github.com/openwrt/uci && cd uci $ wget -q https://github.com/openwrt/some-project/raw/master/Makefile -O Makefile.ci $ make ci-prepare -f Makefile.ci $ podman run --rm --tty --interactive --volume $(pwd):/home/build/openwrt \ ghcr.io/openwrt/ci-tools/native-testing/clang14_snapshot:20230305 \ make ci-native-build -f Makefile.ci Have a nice weekend. Cheers, Petr
Hi Petr, thanks for the comments. These patches are my minimal version to get something running. I will try to extend it in the new few weeks. On 3/11/23 09:57, Petr Štetiar wrote: > Hauke Mehrtens <hauke@hauke-m.de> [2023-03-09 00:18:10]: > > Hi, > > thanks for taking care, LGTM for a start. > > I'll just provide my past experience, something to consider as we're likely > going to bump into those in the long term, so ideally take them into the > account in the long term. > >> clang 14 generates debug information in DWARF 5 format, but valgrind >> 19.0 does not support that. Install valgrind 20.0 from experimental >> which supports it. > > IMO we should aim for reproducible results, thus we should likely provide tools > containers with a known versions inside, so anyone is able to get same results > using the source code Git hash and tool container version. Yes that is a good idea. Debian bookworm is now in code freeze phase so it should not change much any more, but having a stable container would be nice. > Your current approach is highly dynamic, so if you're lucky enough, you might > get a green CI light in the PR branch as the pipeline was run for example 7 > days ago, so you're going to merge it into the upstream branch, but then it's > going to fail as meanwhile Debian has bumped several packages and you're going > to get a CI pipeline failure. > > IMO there should be a tools container Git repository, where we could build, > test and provide versioned containers, for example: > > ghcr.io/openwrt/ci-tools/native-testing/clang14:20230305 > ghcr.io/openwrt/ci-tools/native-testing/clang15:20230305 > ghcr.io/openwrt/ci-tools/native-testing/gcc-8:20230305 > ghcr.io/openwrt/ci-tools/native-testing/gcc-11:20230305 > ghcr.io/openwrt/ci-tools/native-testing/gcc-12:20230305 I will look into this. > We want to test with with multiple compiler versions as those tested changes > might be backported into stable branches, we should even consider to have > stable branches in every project so we could CI test them there as well, folks > would simply create a backport PR in the respective project. Often we only create a stable branch when we really need it in OpenWrt, many of these repositories do not have one. If we create a stable branch we should also have CI on it. > >> +++ b/.github/workflows/test.yml >> @@ -0,0 +1,83 @@ >> +name: 'Run tests' > > We've like 30+ C projects which we should likely cover in the end, thus > considering possible additional 2 stable branches in each, it's around 60+ of > similar workflow files duplicated in various repositories. > > I would consider this future maintenance overhead (imagine just a simple > change or a fix being propagated into 60+ repositories/branches), so I would > recommend to create a shareable Github Action instead: > > uses: openwrt/gh-actions-ci-native > env: > CI_NATIVE_TOOLCHAIN: clang14 > > uses: openwrt/gh-actions-ci-sdk > env: > CI_TARGET_SDK_RELEASE: master > CI_TARGET_SDK: ath79-generic > CI_TARGET_BUILD_DEPENDS: uci > > You can take a look at the ucode project for a very dated, but still working > GitHub example, some bits are even present in uci repsitory in the .gitlab-ci.yml > file. I agree with you that we will have a lot of code duplication in the way I proposed it now. I will have a look at this. > >> + - name: Checkout libubox >> + uses: actions/checkout@v3 >> + with: >> + repository: openwrt/libubox >> + path: libubox > > This looks like another source of unreliability, similar to the toolchain > versions above. For the start, I would simply include all those dependencies > in the native toolchain container, thus we would need to have separate > containers for each branch: If the API changes we would have to update the containers too. I would prefer to use master of all components or even check if there is a branch with the same name and use that one. I think we recently had some changes to iwinfo and then some changes in rpcd which depended on that. I would like to support such changes too. > ghcr.io/openwrt/ci-tools/native-testing/clang14_snapshot:20230305 > ghcr.io/openwrt/ci-tools/native-testing/clang14_openwrt-22.03:20230305 > ghcr.io/openwrt/ci-tools/native-testing/clang14_openwrt-21.02:20230305 > > ghcr.io/openwrt/ci-tools/native-testing/distro_snapshot:20230305 (distro default gcc12) > ghcr.io/openwrt/ci-tools/native-testing/distro_21.02:20230305 (gcc8) > ghcr.io/openwrt/ci-tools/native-testing/distro_22.03:20230305 (gcc11) > > So in the end, ideally, interested developer can have the same CI > failure/result locally and debug/fix it faster: > > $ git clone https://github.com/openwrt/uci && cd uci > $ wget -q https://github.com/openwrt/some-project/raw/master/Makefile -O Makefile.ci > $ make ci-prepare -f Makefile.ci > $ podman run --rm --tty --interactive --volume $(pwd):/home/build/openwrt \ > ghcr.io/openwrt/ci-tools/native-testing/clang14_snapshot:20230305 \ > make ci-native-build -f Makefile.ci > > Have a nice weekend. > > Cheers, > > Petr Hauke
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..d76300d --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,83 @@ +name: 'Run tests' + +on: + push: + +jobs: + build_test: + strategy: + matrix: + include: + - compiler: "-DCMAKE_C_COMPILER=clang" + - compiler: "-DCMAKE_C_COMPILER=gcc" + + name: Build and run tests + runs-on: ubuntu-latest + container: + image: debian:bookworm + + steps: + + - name: Add Debian experimental + run: | + echo "deb http://deb.debian.org/debian/ experimental main" >> /etc/apt/sources.list + + - name: Install compiler + run: | + apt update + apt install -y cmake build-essential lua5.1 pkgconf libjson-c-dev liblua5.1-0-dev python3.11-venv clang libc++abi-dev libc++-dev + apt -t experimental install -y valgrind + useradd -ms /bin/bash buildbot + + - name: Checkout libubox + uses: actions/checkout@v3 + with: + repository: openwrt/libubox + path: libubox + + - name: Checkout uci + uses: actions/checkout@v3 + with: + path: uci + + - name: Create build directory + run: mkdir libubox-build + + - name: Create build directory + run: mkdir uci-build + + - name: Create install directory + run: mkdir install + + - name: Fix permission + run: chown -R buildbot:buildbot libubox-build libubox uci uci-build install + + - name: Run cmake (libubox) + shell: su buildbot -c "sh -e {0}" + working-directory: libubox-build + run: cmake ../libubox -DCMAKE_INSTALL_PREFIX=../install -DBUILD_LUA=false ${{ matrix.compiler }} + + - name: Run build (libubox) + shell: su buildbot -c "sh -e {0}" + working-directory: libubox-build + run: make + + - name: Run install (libubox) + shell: su buildbot -c "sh -e {0}" + working-directory: libubox-build + run: make install + + - name: Run cmake (uci) + shell: su buildbot -c "sh -e {0}" + working-directory: uci-build + run: cmake ../uci -DUNIT_TESTING=true -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_PREFIX_PATH=../install ${{ matrix.compiler }} + + - name: Run build (uci) + shell: su buildbot -c "sh -e {0}" + working-directory: uci-build + run: make + + - name: Run tests (uci) + shell: su buildbot -c "sh -e {0}" + working-directory: uci-build + run: make test
Add a github action to build uci and then execute the tests. This first builds and installs libubox and then uses that dependency to build and test uci. clang 14 generates debug information in DWARF 5 format, but valgrind 19.0 does not support that. Install valgrind 20.0 from experimental which supports it. Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de> --- I created a github pull request with these changes too: https://github.com/openwrt/libubox/pull/2 .github/workflows/test.yml | 83 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 .github/workflows/test.yml