From patchwork Tue Dec 1 17:18:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= X-Patchwork-Id: 1408939 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=S4HIHE05; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4ClpkC47hGz9sW4 for ; Wed, 2 Dec 2020 04:19:31 +1100 (AEDT) Received: from localhost ([::1]:41212 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kk9JJ-0003xD-HW for incoming@patchwork.ozlabs.org; Tue, 01 Dec 2020 12:19:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35664) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kk9IU-0003wt-1t for qemu-devel@nongnu.org; Tue, 01 Dec 2020 12:18:38 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:23294) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kk9IO-00021t-VI for qemu-devel@nongnu.org; Tue, 01 Dec 2020 12:18:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1606843112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zBwfNvXJQOTT+gigvmBQJupbV6P0/JditukNZ0ssaTM=; b=S4HIHE05q/zEx2qSKcgGRPbvXuNnpK1RuQlFGgw4wMdwe7Bn3MwafepKYwA5T2B6PvfI5j y0rAyicdKnl89b6HNURCw64Jw2CafqXm1GLsMSjneqiOYV2xkXKN2DobQ5pD7YUrL3OX4Q v6c+oYPP4QGt0i2qiiJG1fC/tBgcxLg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-586-E7Mrk6D8Or-GBndOIhX-iQ-1; Tue, 01 Dec 2020 12:18:30 -0500 X-MC-Unique: E7Mrk6D8Or-GBndOIhX-iQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B54BB185E483; Tue, 1 Dec 2020 17:18:28 +0000 (UTC) Received: from fedora.redhat.com (ovpn-115-4.ams2.redhat.com [10.36.115.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24FCC5D9C2; Tue, 1 Dec 2020 17:18:26 +0000 (UTC) From: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= To: qemu-devel@nongnu.org Subject: [PATCH 00/23] tests/docker: start using libvirt-ci's "lcitool" for dockerfiles Date: Tue, 1 Dec 2020 17:18:02 +0000 Message-Id: <20201201171825.2243775-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=berrange@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -35 X-Spam_score: -3.6 X-Spam_bar: --- X-Spam_report: (-3.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.497, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , =?utf-8?q?Alex?= =?utf-8?q?_Benn=C3=A9e?= , =?utf-8?q?Daniel_P=2E_Be?= =?utf-8?q?rrang=C3=A9?= , Peter Maydell Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Currently the tests/docker/dockerfiles/*Dockerfile recipes are all hand written by contributors. There is a common design pattern, but the set of packages listed for installation leaves alot to be desired - There is no consistency at all across distros - Many potential build deps are not listed in the containers - Some packages are not used by QEMU at all - Adding new distros is an error prone task The same applies to package lists for VMs, Cirrus CI / Travis CI, and probably more. This problem is not unique to QEMU, libvirt faced the exact same issues and developed a program called "lcitool" which is part of the libvirt-ci git repository to reduce the burden in this area. https://gitlab.com/libvirt-ci/libvirt-ci/ Despite its name, this repository is not tied to libvirt, and so as well as the 40+ libvirt git repos, it is also used by the libosinfo and virt-viewer projects for their CI needs. The idea is that all these projects can share the burden of libvirt-ci. lcitool is capable of automating the installation and updating of VM images, creation of dockerfiles and creation of standalone package lists. In this series I'm taking the easy step which is the generation of dockerfiles, since that is also where the most immediate value lies for QEMU. The key concept in lcitool that brings a huge win in maintainability is that there is a single file which defines a mapping between a build pre-requisite and the native package on each targetted distro. https://gitlab.com/berrange/libvirt-ci/-/blob/qemu/guests/lcitool/lcitool/= ansible/vars/mappings.yml A project merely has to have its list of pre-requisites enumerated https://gitlab.com/berrange/libvirt-ci/-/blob/qemu/guests/lcitool/lcitool/a= nsible/vars/projects/qemu.yml The combination of these two files is enough to generate accurate package lists for any supported distro. Currently supported distros are Debian (10, sid), Ubuntu (18.04, 20.04), CentOS (7, 8, stream), Fedora (32, 33, rawhide), OpenSUSE (151) macOS (homebrew), FreeBSD (11, 12, current). At the end of this series, I have dockerfiles auto-generated for QEMU covering Ubuntu 18.04 & 20.04, CentOS 7 & 8 and Fedora 32. lcitool is also capable of generating dockerfiles for cross-compiled non-x86 architectures for Debian, and for mingw32/64 for Fedora. This is driven from the very same mapping.yml file listed above, which has attributes to indicate whether a given dependancy should be pulled from the native or cross build target. Again this means that we have strong guarantee of consistent deps being used between cross containers. I have not converted cross containers in this series though, because the way we generated cross dockerfiles is different from how QEMU does it. lcitool will always generate fully self-contained dockerfiles, but QEMU currently uses layered dockerfiles for cross-builds, so all cross builds share a common intermediate container. I could enhance lcitool to support layered containers for cross-builds, but before doing that I wondered how strongly people are attached to them ? If self-contained dockerfiles are acceptable I can do that more easily. There is also scope for auto-generating the package lists for tests/vm and .cirrus.yml files, but I've not attempted that here. The same general idea appies - we just call lcitool to spit out a list of native packages for each case. If converting tests/vm, we would need to add more distros to lcitool mappings.yml to covert openbsd, netbsd, haiku since libvirt does not target those distros itself. Finally I wondered about the approach to integrating with lcitool. I have provided a tests/docker/dockerfiles/refresh script that needs to be invoked periodically to re-generate them. eg when adding a new distro, or when the package lists change. I could add libvirt-ci.git as a sub-module and provide a more seemless integration, but open to suggestions. In libvirt*.git repos we don't bother with git submodules for libvirt-ci.git since whomever runs it to refresh containers just has a local checkout regardless. Note since this is a proof of concept, the additions to libvirt-ci for QEMU are not part of the main git repo yet, they're just in my own fork on the "qemu" branch https://gitlab.com/berrange/libvirt-ci/-/tree/qemu Daniel P. Berrang=C3=A9 (23): hw/usb/ccid: remove references to NSS tests/docker: don't use BUILDKIT in GitLab either tests/docker: use project specific container registries tests/docker: use explicit docker.io registry tests/docker: remove travis container tests/docker: remove FEATURES env var from templates tests/docker: fix sorting in package lists tests/docker: fix mistakes in centos package lists tests/docker: fix mistakes in fedora package list tests/docker: fix mistakes in ubuntu package lists tests/docker: remove mingw packages from Fedora tests/docker: add script for automating container refresh tests/docker: expand centos7 package list tests/docker: expand centos8 package list tests/docker: expand fedora package list tests/docker: expand ubuntu1804 package list tests/docker: expand ubuntu2004 package list tests/docker: auto-generate centos7 with lcitool tests/docker: auto-generate centos8 with lcitool tests/docker: auto-generate fedora with lcitool tests/docker: auto-generate ubuntu1804 with lcitool tests/docker: auto-generate ubuntu2004 with lcitool tests/docker: remove ubuntu container .gitlab-ci.d/containers.yml | 5 - .travis.yml | 14 +- docs/ccid.txt | 15 +- scripts/coverity-scan/coverity-scan.docker | 1 - tests/docker/common.rc | 19 +- tests/docker/docker.py | 4 +- tests/docker/dockerfiles/centos7.docker | 157 +++++++++--- tests/docker/dockerfiles/centos8.docker | 153 ++++++++--- .../dockerfiles/debian-xtensa-cross.docker | 2 +- tests/docker/dockerfiles/debian10.docker | 4 +- tests/docker/dockerfiles/debian11.docker | 2 +- .../dockerfiles/fedora-cris-cross.docker | 2 +- .../dockerfiles/fedora-i386-cross.docker | 2 +- .../dockerfiles/fedora-win32-cross.docker | 3 +- .../dockerfiles/fedora-win64-cross.docker | 3 +- tests/docker/dockerfiles/fedora.docker | 241 ++++++++++-------- tests/docker/dockerfiles/refresh | 67 +++++ tests/docker/dockerfiles/travis.docker | 17 -- tests/docker/dockerfiles/ubuntu.docker | 71 ------ tests/docker/dockerfiles/ubuntu1804.docker | 185 +++++++++----- tests/docker/dockerfiles/ubuntu2004.docker | 197 +++++++++----- tests/docker/run | 3 - tests/docker/test-clang | 2 +- tests/docker/test-debug | 2 +- tests/docker/test-mingw | 3 +- tests/docker/test-misc | 2 +- tests/docker/test-tsan | 2 +- tests/docker/travis | 22 -- tests/docker/travis.py | 47 ---- 29 files changed, 732 insertions(+), 515 deletions(-) create mode 100755 tests/docker/dockerfiles/refresh delete mode 100644 tests/docker/dockerfiles/travis.docker delete mode 100644 tests/docker/dockerfiles/ubuntu.docker delete mode 100755 tests/docker/travis delete mode 100755 tests/docker/travis.py --=20 2.28.0