From patchwork Thu Mar 4 03:09:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniele Buono X-Patchwork-Id: 1447057 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=sboUS/Y5; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DrbVf4LKTz9sSC for ; Thu, 4 Mar 2021 14:11:22 +1100 (AEDT) Received: from localhost ([::1]:52430 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHeOW-0006mw-KD for incoming@patchwork.ozlabs.org; Wed, 03 Mar 2021 22:11:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:52960) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHeND-0005u6-UT for qemu-devel@nongnu.org; Wed, 03 Mar 2021 22:09:59 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:59388) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHeNC-0005Ja-AU for qemu-devel@nongnu.org; Wed, 03 Mar 2021 22:09:59 -0500 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 12434nB4141633; Wed, 3 Mar 2021 22:09:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=rlWKVEfsQDO2b7TeAaKs8qinmav8DqPFG1StZI6mWBU=; b=sboUS/Y5kRx9jRPJFtDj7CjmkygBoFlE0iozfUulsmZHNhwEXCN6PqR80KH4aTuELF1/ ZsPeuFtAvSC788WzXNU2QExHLZzK01mhcSi98onjpERo72cXE8cvK5UyBr2WF/VN5nZb i05rNU7EvzXzoceafpD6xnhU8llDhwiWDKwT0j5bZ8puL1AD2okc4+p36hhcmc2CbW4k c5NMNKJKiSlMfUjWZ3oO8caTj5UAIyf6eAZmU9CrINAGTwjJqhvcX6+gD/K/ZLTJqcwJ w9kbjsvHTAmmwuEhjup5OxMwbULNM/wOs/6TCUOQ/qqA4BGqyBCH1lDxeoQFEmKh9IId xg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 372q8b86t4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Mar 2021 22:09:55 -0500 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 12435rvp146438; Wed, 3 Mar 2021 22:09:55 -0500 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 372q8b86su-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Mar 2021 22:09:55 -0500 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 12438bLM023154; Thu, 4 Mar 2021 03:09:54 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma02dal.us.ibm.com with ESMTP id 3710sr1py5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 04 Mar 2021 03:09:54 +0000 Received: from b03ledav003.gho.boulder.ibm.com (b03ledav003.gho.boulder.ibm.com [9.17.130.234]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 12439r5J42926534 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 4 Mar 2021 03:09:53 GMT Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3ABF36A04F; Thu, 4 Mar 2021 03:09:53 +0000 (GMT) Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8B7806A047; Thu, 4 Mar 2021 03:09:52 +0000 (GMT) Received: from Buonos-Thinkpad-X1.ibm.com (unknown [9.211.86.147]) by b03ledav003.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 4 Mar 2021 03:09:52 +0000 (GMT) From: Daniele Buono To: qemu-devel@nongnu.org Subject: [PATCH v3 1/2] gitlab-ci.yml: Allow custom # of parallel linkers Date: Wed, 3 Mar 2021 22:09:47 -0500 Message-Id: <20210304030948.9367-2-dbuono@linux.vnet.ibm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210304030948.9367-1-dbuono@linux.vnet.ibm.com> References: <20210304030948.9367-1-dbuono@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-04_01:2021-03-03, 2021-03-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 adultscore=0 malwarescore=0 bulkscore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 suspectscore=0 mlxlogscore=999 spamscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103040008 Received-SPF: none client-ip=148.163.156.1; envelope-from=dbuono@linux.vnet.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , =?utf-8?q?Phi?= =?utf-8?q?lippe_Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Paolo Bonzini , =?utf-8?q?Alex_Benn=C3=A9e?= , Daniele Buono Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Define a new variable LD_JOBS, that can be used to select the maximum number of linking jobs to be executed in parallel. If the variable is not defined, maintain the default given by make -j Currently, make parallelism at build time is based on the number of cpus available. This doesn't work well with LTO at linking, because with LTO the linker has to load in memory all the intermediate object files for optimization. The end result is that, if the gitlab runner happens to run two linking processes at the same time, the job will fail with an out-of-memory error, This patch leverages the ability to maintain high parallelism at compile time, but limit the number of linkers executed in parallel. Signed-off-by: Daniele Buono Reviewed-by: Daniel P. Berrangé --- .gitlab-ci.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 8b6d495288..814f51873f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -27,6 +27,10 @@ include: else ../configure --enable-werror $CONFIGURE_ARGS ; fi || { cat config.log meson-logs/meson-log.txt && exit 1; } + - if test -n "$LD_JOBS"; + then + meson configure . -Dbackend_max_links="$LD_JOBS" ; + fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS"; then From patchwork Thu Mar 4 03:09:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniele Buono X-Patchwork-Id: 1447058 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=sYkAczNu; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DrbXR1HM1z9sPf for ; Thu, 4 Mar 2021 14:12:55 +1100 (AEDT) Received: from localhost ([::1]:54582 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHeQ1-0007m9-6o for incoming@patchwork.ozlabs.org; Wed, 03 Mar 2021 22:12:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:52962) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHeNF-0005vc-Ay for qemu-devel@nongnu.org; Wed, 03 Mar 2021 22:10:01 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:10904) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHeNC-0005K8-E8 for qemu-devel@nongnu.org; Wed, 03 Mar 2021 22:10:01 -0500 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 12432uMX148366; Wed, 3 Mar 2021 22:09:57 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=TqZAcUMpsNrYgZ/xK7DqYuBIgHFqepyQHDa7X5yETjE=; b=sYkAczNuYIpDrk2qVYGQDWMcojTDqZOUf01/nk1vf87GgBeRIi5af8QLcJ50VEwvaeSX lj8hf6SRRdTIPxzlZsh+jCff6OpzO50YAZ7wzpHXDmHwaTODC76VtOvs+SBQgte8Q+o3 J0yizhQVhVF+TVke/nQAjZqPWhcsjN/FyGpJHRTEi6vzsvp/bzdNZjLpA77Mdyu8nG/F K8zD/ENi+tMIb8eeJeJVXdMFiuT7mQCVN42K//wjo1LblkVjBeW2T5YK6ANWAkOjZbWi PXdaqqyPO61smX8HpvHdugiWnK0apYTA1mbuzz7nqy+fAcRuSv9Zj45zpjwT9fzeXw1K xw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 372mtm3h8p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Mar 2021 22:09:57 -0500 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 12434WDj157932; Wed, 3 Mar 2021 22:09:56 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 372mtm3h8d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Mar 2021 22:09:56 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 124371Ng002105; Thu, 4 Mar 2021 03:09:56 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma01dal.us.ibm.com with ESMTP id 371qmupts0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 04 Mar 2021 03:09:56 +0000 Received: from b03ledav003.gho.boulder.ibm.com (b03ledav003.gho.boulder.ibm.com [9.17.130.234]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 12439sja20250936 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 4 Mar 2021 03:09:55 GMT Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D7BD66A047; Thu, 4 Mar 2021 03:09:54 +0000 (GMT) Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3486A6A04D; Thu, 4 Mar 2021 03:09:54 +0000 (GMT) Received: from Buonos-Thinkpad-X1.ibm.com (unknown [9.211.86.147]) by b03ledav003.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 4 Mar 2021 03:09:54 +0000 (GMT) From: Daniele Buono To: qemu-devel@nongnu.org Subject: [PATCH v3 2/2] gitlab-ci.yml: Add jobs to test CFI flags Date: Wed, 3 Mar 2021 22:09:48 -0500 Message-Id: <20210304030948.9367-3-dbuono@linux.vnet.ibm.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210304030948.9367-1-dbuono@linux.vnet.ibm.com> References: <20210304030948.9367-1-dbuono@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-04_01:2021-03-03, 2021-03-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 adultscore=0 mlxscore=0 priorityscore=1501 impostorscore=0 clxscore=1015 bulkscore=0 malwarescore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103040008 Received-SPF: none client-ip=148.163.158.5; envelope-from=dbuono@linux.vnet.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , =?utf-8?q?Phi?= =?utf-8?q?lippe_Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Paolo Bonzini , =?utf-8?q?Alex_Benn=C3=A9e?= , Daniele Buono Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" QEMU has had options to enable control-flow integrity features for a few months now. Add two sets of build/check/acceptance jobs to ensure the binary produced is working fine. The three sets allow testing of x86_64 binaries for x86_64, s390x, ppc64 and aarch64 targets Signed-off-by: Daniele Buono --- .gitlab-ci.yml | 119 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 814f51873f..7b1f25c92e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -483,6 +483,125 @@ clang-user: --extra-cflags=-fsanitize=undefined --extra-cflags=-fno-sanitize-recover=undefined MAKE_CHECK_ARGS: check-unit check-tcg +# Set LD_JOBS=1 because this requires LTO and ld consumes a large amount of memory. +# On gitlab runners, default value sometimes end up calling 2 lds concurrently and +# triggers an Out-Of-Memory error +# +# Since slirp callbacks are used in QEMU Timers, slirp needs to be compiled together +# with QEMU and linked as a static library to avoid false positives in CFI checks. +# This can be accomplished by using -enable-slirp=git, which avoids the use of +# a system-wide version of the library +# +# Split in three sets of build/check/acceptance to limit the execution time of each +# job +build-cfi-arm: + <<: *native_build_job_definition + needs: + - job: amd64-fedora-container + variables: + LD_JOBS: 1 + AR: llvm-ar + IMAGE: fedora + CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug + --enable-safe-stack --enable-slirp=git + TARGETS: aarch64-softmmu + MAKE_CHECK_ARGS: check-build + artifacts: + expire_in: 2 days + paths: + - build + +check-cfi-arm: + <<: *native_test_job_definition + needs: + - job: build-cfi-arm + artifacts: true + variables: + IMAGE: fedora + MAKE_CHECK_ARGS: check + +acceptance-cfi-arm: + <<: *native_test_job_definition + needs: + - job: build-cfi-arm + artifacts: true + variables: + IMAGE: fedora + MAKE_CHECK_ARGS: check-acceptance + <<: *acceptance_definition + +build-cfi-ibm: + <<: *native_build_job_definition + needs: + - job: amd64-fedora-container + variables: + LD_JOBS: 1 + AR: llvm-ar + IMAGE: fedora + CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug + --enable-safe-stack --enable-slirp=git + TARGETS: ppc64-softmmu s390x-softmmu + MAKE_CHECK_ARGS: check-build + artifacts: + expire_in: 2 days + paths: + - build + +check-cfi-ibm: + <<: *native_test_job_definition + needs: + - job: build-cfi-ibm + artifacts: true + variables: + IMAGE: fedora + MAKE_CHECK_ARGS: check + +acceptance-cfi-ibm: + <<: *native_test_job_definition + needs: + - job: build-cfi-ibm + artifacts: true + variables: + IMAGE: fedora + MAKE_CHECK_ARGS: check-acceptance + <<: *acceptance_definition + +build-cfi-intel: + <<: *native_build_job_definition + needs: + - job: amd64-fedora-container + variables: + LD_JOBS: 1 + AR: llvm-ar + IMAGE: fedora + CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug + --enable-safe-stack --enable-slirp=git + TARGETS: x86_64-softmmu + MAKE_CHECK_ARGS: check-build + artifacts: + expire_in: 2 days + paths: + - build + +check-cfi-intel: + <<: *native_test_job_definition + needs: + - job: build-cfi-intel + artifacts: true + variables: + IMAGE: fedora + MAKE_CHECK_ARGS: check + +acceptance-cfi-intel: + <<: *native_test_job_definition + needs: + - job: build-cfi-intel + artifacts: true + variables: + IMAGE: fedora + MAKE_CHECK_ARGS: check-acceptance + <<: *acceptance_definition + tsan-build: <<: *native_build_job_definition variables: