[{"id":3678948,"web_url":"http://patchwork.ozlabs.org/comment/3678948/","msgid":"","list_archive_url":null,"date":"2026-04-18T06:29:34","subject":"Re: [PATCH] testsuite: flag failure to detect failures","submitter":{"id":91428,"url":"http://patchwork.ozlabs.org/api/people/91428/","name":"Andrew Pinski","email":"andrew.pinski@oss.qualcomm.com"},"content":"On Fri, Apr 17, 2026 at 11:24 PM Alexandre Oliva wrote:\n>\n>\n> There are various ways for tests to report an execution failure.\n>\n> Make sure the testing infrastructure can detect all of them, so that\n> failures don't go silent.\n>\n> Regstrapping on x86_64-linux-gnu... Already tested on a target where we\n> failed to detect a negative exit status as a failure, only noticed\n> because g++.dg/contracts/contracts-ignore2.C's execution failure was not\n> detected as such. We'd better have stronger defenses against failure to\n> detect failures. Ok to install?\n\nOk for stage 1. Even though this won't break anything except add a few\ntestcases failures for some bad dejagnu boards; I rather wait because\nwe are just a few days before branching.\n\nThanks,\nAndrew\n\n>\n>\n> for gcc/testsuite/ChangeLog\n>\n> * gcc.dg/shouldfail-abort.c: New.\n> * gcc.dg/shouldfail-exit-neg.c: New.\n> * gcc.dg/shouldfail-exit-pos.c: New.\n> * gcc.dg/shouldfail-return-neg.c: New.\n> * gcc.dg/shouldfail-return-pos.c: New.\n> ---\n> gcc/testsuite/gcc.dg/shouldfail-abort.c | 10 ++++++++++\n> gcc/testsuite/gcc.dg/shouldfail-exit-neg.c | 10 ++++++++++\n> gcc/testsuite/gcc.dg/shouldfail-exit-pos.c | 10 ++++++++++\n> gcc/testsuite/gcc.dg/shouldfail-return-neg.c | 10 ++++++++++\n> gcc/testsuite/gcc.dg/shouldfail-return-pos.c | 10 ++++++++++\n> 5 files changed, 50 insertions(+)\n> create mode 100644 gcc/testsuite/gcc.dg/shouldfail-abort.c\n> create mode 100644 gcc/testsuite/gcc.dg/shouldfail-exit-neg.c\n> create mode 100644 gcc/testsuite/gcc.dg/shouldfail-exit-pos.c\n> create mode 100644 gcc/testsuite/gcc.dg/shouldfail-return-neg.c\n> create mode 100644 gcc/testsuite/gcc.dg/shouldfail-return-pos.c\n>\n> diff --git a/gcc/testsuite/gcc.dg/shouldfail-abort.c b/gcc/testsuite/gcc.dg/shouldfail-abort.c\n> new file mode 100644\n> index 0000000000000..a00c483a56ff0\n> --- /dev/null\n> +++ b/gcc/testsuite/gcc.dg/shouldfail-abort.c\n> @@ -0,0 +1,10 @@\n> +/* { dg-do run } */\n> +/* { dg-shouldfail \"\" } */\n> +\n> +/* Raise red flags if the way programs are meant to flag failures aren't\n> + detected as such. This is unlikely to indicate a compiler problem, but\n> + maybe a runtime or test infrastructure problem. */\n> +\n> +int main() {\n> + __builtin_abort();\n> +}\n> diff --git a/gcc/testsuite/gcc.dg/shouldfail-exit-neg.c b/gcc/testsuite/gcc.dg/shouldfail-exit-neg.c\n> new file mode 100644\n> index 0000000000000..57d66814120a6\n> --- /dev/null\n> +++ b/gcc/testsuite/gcc.dg/shouldfail-exit-neg.c\n> @@ -0,0 +1,10 @@\n> +/* { dg-do run } */\n> +/* { dg-shouldfail \"\" } */\n> +\n> +/* Raise red flags if the way programs are meant to flag failures aren't\n> + detected as such. This is unlikely to indicate a compiler problem, but\n> + maybe a runtime or test infrastructure problem. */\n> +\n> +int main() {\n> + __builtin_exit(-1);\n> +}\n> diff --git a/gcc/testsuite/gcc.dg/shouldfail-exit-pos.c b/gcc/testsuite/gcc.dg/shouldfail-exit-pos.c\n> new file mode 100644\n> index 0000000000000..2ae16fa2879ff\n> --- /dev/null\n> +++ b/gcc/testsuite/gcc.dg/shouldfail-exit-pos.c\n> @@ -0,0 +1,10 @@\n> +/* { dg-do run } */\n> +/* { dg-shouldfail \"\" } */\n> +\n> +/* Raise red flags if the way programs are meant to flag failures aren't\n> + detected as such. This is unlikely to indicate a compiler problem, but\n> + maybe a runtime or test infrastructure problem. */\n> +\n> +int main() {\n> + __builtin_exit(1);\n> +}\n> diff --git a/gcc/testsuite/gcc.dg/shouldfail-return-neg.c b/gcc/testsuite/gcc.dg/shouldfail-return-neg.c\n> new file mode 100644\n> index 0000000000000..6d32d260ad5c4\n> --- /dev/null\n> +++ b/gcc/testsuite/gcc.dg/shouldfail-return-neg.c\n> @@ -0,0 +1,10 @@\n> +/* { dg-do run } */\n> +/* { dg-shouldfail \"\" } */\n> +\n> +/* Raise red flags if the way programs are meant to flag failures aren't\n> + detected as such. This is unlikely to indicate a compiler problem, but\n> + maybe a runtime or test infrastructure problem. */\n> +\n> +int main() {\n> + return -1;\n> +}\n> diff --git a/gcc/testsuite/gcc.dg/shouldfail-return-pos.c b/gcc/testsuite/gcc.dg/shouldfail-return-pos.c\n> new file mode 100644\n> index 0000000000000..ceb4f4cf95a75\n> --- /dev/null\n> +++ b/gcc/testsuite/gcc.dg/shouldfail-return-pos.c\n> @@ -0,0 +1,10 @@\n> +/* { dg-do run } */\n> +/* { dg-shouldfail \"\" } */\n> +\n> +/* Raise red flags if the way programs are meant to flag failures aren't\n> + detected as such. This is unlikely to indicate a compiler problem, but\n> + maybe a runtime or test infrastructure problem. */\n> +\n> +int main() {\n> + return 1;\n> +}\n>\n> --\n> Alexandre Oliva, happy hacker https://blog.lx.oliva.nom.br/\n> Free Software Activist FSFLA co-founder GNU Toolchain Engineer\n> More tolerance and less prejudice are key for inclusion and diversity.\n> Excluding neuro-others for not behaving \"\"normal\"\" is *not* inclusive!","headers":{"Return-Path":"","X-Original-To":["incoming@patchwork.ozlabs.org","gcc-patches@gcc.gnu.org"],"Delivered-To":["patchwork-incoming@legolas.ozlabs.org","gcc-patches@gcc.gnu.org"],"Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=qualcomm.com header.i=@qualcomm.com header.a=rsa-sha256\n header.s=qcppdkim1 header.b=U8CJmDyR;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com\n header.a=rsa-sha256 header.s=google header.b=DM4HcSqc;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=temperror (SPF Temporary Error: DNS Timeout) smtp.mailfrom=gcc.gnu.org\n (client-ip=2620:52:6:3111::32; helo=vm01.sourceware.org;\n envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org;\n receiver=patchwork.ozlabs.org)","sourceware.org;\n\tdkim=pass (2048-bit key,\n unprotected) header.d=qualcomm.com header.i=@qualcomm.com header.a=rsa-sha256\n header.s=qcppdkim1 header.b=U8CJmDyR;\n\tdkim=pass (2048-bit key,\n unprotected) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com\n header.a=rsa-sha256 header.s=google header.b=DM4HcSqc","sourceware.org; dmarc=none (p=none dis=none)\n header.from=oss.qualcomm.com","sourceware.org;\n spf=pass smtp.mailfrom=oss.qualcomm.com","server2.sourceware.org;\n arc=pass smtp.remote-ip=205.220.168.131"],"Received":["from vm01.sourceware.org (vm01.sourceware.org\n [IPv6:2620:52:6:3111::32])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4fyMM42163z1yGt\n\tfor ; Sat, 18 Apr 2026 16:30:19 +1000 (AEST)","from vm01.sourceware.org (localhost [127.0.0.1])\n\tby sourceware.org (Postfix) with ESMTP id 58F4C4A9C029\n\tfor ; Sat, 18 Apr 2026 06:30:17 +0000 (GMT)","from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com\n [205.220.168.131])\n by sourceware.org (Postfix) with ESMTPS id C044F4AA51F8\n for ; Sat, 18 Apr 2026 06:29:48 +0000 (GMT)","from pps.filterd (m0279862.ppops.net [127.0.0.1])\n by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id\n 63I4UGAV1410479\n for ; Sat, 18 Apr 2026 06:29:46 GMT","from mail-dy1-f197.google.com (mail-dy1-f197.google.com\n [74.125.82.197])\n by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4dm2qag6wf-1\n (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT)\n for ; Sat, 18 Apr 2026 06:29:46 +0000 (GMT)","by mail-dy1-f197.google.com with SMTP id\n 5a478bee46e88-2d93379001eso4552453eec.1\n for ; Fri, 17 Apr 2026 23:29:46 -0700 (PDT)"],"DKIM-Filter":["OpenDKIM Filter v2.11.0 sourceware.org 58F4C4A9C029","OpenDKIM Filter v2.11.0 sourceware.org C044F4AA51F8"],"DMARC-Filter":"OpenDMARC Filter v1.4.2 sourceware.org C044F4AA51F8","ARC-Filter":"OpenARC Filter v1.0.0 sourceware.org C044F4AA51F8","ARC-Seal":["i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1776493789; cv=pass;\n b=L+zIOtHSmTbuyD2S2fP5DSWfjsalm+4wPQBAihy4oJFHSc+ltxeOFVa35iarBevKP9iYcFjqh4XKlvjVSiag0xE7R9eC50QO39PL98K+2kkm1AHCURE/Neako1eIEhuCDVNs05VPh6ka/jwzC2p7oPJeFRxhgqK34WeMF71n70s=","i=1; a=rsa-sha256; t=1776493786; cv=none;\n d=google.com; s=arc-20240605;\n b=cUHhYix8DC2OOeGsAEFRMXG6mHrMMa7GeOjf0kncG0GNLC55QgTTxUQNZ77aMntdrG\n lCDwo2QLkAsz4Ga76syIxo/PI5sJAIzUWnwiRzwF5VVDMgyRINQ/6Je38ImiG3QQeRrO\n tNX/fKVNj6stV0Pj4l6b36qTf7YQhAHi4/zt7/94fOyx5WiSlDCuvnO/T+zsdQ0ADV5K\n SPxZ9osLFBPLm+OhGPNLuEzgzuPaB158T+JrNmG1PCNc2vaFkXHqbW/Dn3wYr5lAKZmE\n /g7ge3GUoQagvIZ2fQWi9BHX6XNKsKZG2o6svlIWElWvy2vI0036SMljBBN4Mpl6ycIb\n rDFg=="],"ARC-Message-Signature":["i=2; a=rsa-sha256; d=sourceware.org; s=key;\n t=1776493789; c=relaxed/simple;\n bh=wtiJL0aKtHuSaq4S7w1PQRoWAuwBkfx2Cxq9RmQShwk=;\n h=DKIM-Signature:DKIM-Signature:MIME-Version:From:Date:Message-ID:\n Subject:To;\n b=cKI5b0walSi7ZTrEmhVU4cC/F58ESEDDNDSeUxxYaDnxDG8yT/U6+NhPvtJEKNLPv/mxk65LzrzKAdKKCu57CA/PMZ71fA/3X/AMqOrOiIZMxpBWrTun1NhNMVI5nMyY+0fnsYs/3KpYFdXei6jOnqU2jM1K3az1N6Z61zOJc9o=","i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;\n s=arc-20240605;\n h=content-transfer-encoding:cc:to:subject:message-id:date:from\n :in-reply-to:references:mime-version:dkim-signature;\n bh=RRe3S8M2NdaSsgdaC7YHmlPv6xitpLPorsMIa4ZcDJo=;\n fh=BIzq9g2EU/r3TGLVkENl2gLZH2d8yI/FW4bSU7YZUKA=;\n b=FKVTzvFPOngS9m7T8N6nvq1QpRf6qRWiQXCgq94GdeiBreLxNkOVtmTnTTSoRw9J3c\n DyHlIOP+JY9qAUcPip7eOgGQS6jUyBJajIol5oL/fSW16Q8Bo9hlrfQCGejiuATROvlE\n Ym9LI+wD1a2I0XWgoAepDDdJ6tnuYTCnH9schYyx9uGmUXII5K2K4eWN/O8gS+No0W2v\n bzzaMWhp/EK2rfdb4C8KMjmAbLfxn/cFaN5/zrDkfWN/4QIr9n0L1v0TbWNCXtRsUTuQ\n SF7h4W6Tfx26CJ2klFab84r+5aQSA6Pec18H58v4rvRZ6cDwBAsa15kXXIFSpNVTNeVR\n K7+Q==; darn=gcc.gnu.org"],"ARC-Authentication-Results":["i=2; server2.sourceware.org","i=1; mx.google.com; arc=none"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h=\n cc:content-transfer-encoding:content-type:date:from:in-reply-to\n :message-id:mime-version:references:subject:to; s=qcppdkim1; bh=\n RRe3S8M2NdaSsgdaC7YHmlPv6xitpLPorsMIa4ZcDJo=; b=U8CJmDyRZsRLxnZx\n AmmXNoYA+/o9aBWviiEVTP18GxB4GQ+9blZjrxsIPtlJWL3lUh+I30bM9iENEIth\n gocNyYNdVZDNFbycHqmfNGA/S7tZnxsoPqjHDq75vSjVPQM3OqI2BSyf/ulIo+iy\n peucrjxLj+8xAJDE52dFcytqPa4h8Hz+uqDc69eQzq1em709bNHqTjHNDbHa7T8U\n K8X5PuTG8cTYTMjh43zU2M9x6JVtnd+I9G+S3Jvdgh9HnyzePX2untSp8Us6rC4x\n 7wlRwPvXiuj49V20WvjJJ7xMmqluu9pGdvzcc3urXzdrGxnrid43fe2vgmynWzrX\n ZDsQKQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oss.qualcomm.com; s=google; t=1776493786; x=1777098586; darn=gcc.gnu.org;\n h=content-transfer-encoding:cc:to:subject:message-id:date:from\n :in-reply-to:references:mime-version:from:to:cc:subject:date\n :message-id:reply-to;\n bh=RRe3S8M2NdaSsgdaC7YHmlPv6xitpLPorsMIa4ZcDJo=;\n b=DM4HcSqcQz7o02hoFEgqUSd5YRimQ4qV6mu6d0sKcdtbz++pTz0WSZTwjdNCTbS0AD\n MLvtMf9+vyiDx26q32GRaVo59L5KbTnL8ecF8rasI6FXNOwjDJmtwwbcat/pDrGeejGW\n fcylas+kt52ShOULnIA12tj0Ht1M7UD53uNmwI7ZGWzGcL449WXVmexS9Uc4MibWerCB\n LQsv9z8zSYeGNQzXKNhWDW+URQWRtlJwhTADsvnfFsLCxd6+DxDKCdXZlckebiqnbuia\n ax18NiVVUWgUJC7hjIcX4Fy/5CNSKXav6LRsNfqOvyD3Ohx8q17RAozOyaR+6+NGNtuK\n tdQw=="],"X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20251104; t=1776493786; x=1777098586;\n h=content-transfer-encoding:cc:to:subject:message-id:date:from\n :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from\n :to:cc:subject:date:message-id:reply-to;\n bh=RRe3S8M2NdaSsgdaC7YHmlPv6xitpLPorsMIa4ZcDJo=;\n b=fJx9/GYRrRPmydxQiwmijEH12BMwXuBtPWEIl3xrl2ItKfjY27vmGT65lOMkEhYZHU\n SDA5qMyXhF0CMJHuPhuMgubsxy0L9fvpD3S9IGT0KnYFWnSruuz5/iw3RM8P5KhzeqWw\n Zf4ilA4LoSIw1MjkONzGoUiQOQaEMgb/FkK+1bydc6Le3Zv0i7bV8IFyMOi5qQ7Dqfhw\n RlHZYhENEYJu7cwDj6Gmjot3G606/0eCQg8Omd/eAnBm8TC76DaryuqLPJOtKz+zvd/J\n Tr6A40Cz0UFRzrY9hxKAaZwtGe1HSLnQJiMwsIJ/sKp00masls1XPUz5SzeWJVHBHgVI\n ye/Q==","X-Gm-Message-State":"AOJu0Ywgw+ObQXodNuqyj4Bd0FeCdhJSWJeYcq0PiuZrbGgr3cUdWhBP\n jTIo+1DLpHhwiCpkQGH/3Bx4KdjHqgFEO+1WfwSQEc+8P1qBSYmQCKbUs9tOzer9iLRINoVGd7p\n wYwYL9jfbnuxpe1SgsCUnFZiVD2VBVFq5Z08D+Vgg48igjufoVIslAiuTxx9Wg80ZB7ZRXwpomG\n BkpfaXql+WKtg70SztNLxhDdP9Q03zb+hjPVeUlNVYMY0=","X-Gm-Gg":"AeBDiet9O2eA99kDkm4FcGMqTw5KpDlN8L0YtvUAQalrK8Svp7HOCPPFREPXjmNOnyV\n v7qApSuPE4gosFcM/9bIr+FPHHNTWMXwqVcInEbkr48y/efZJ7MLdHtR5dXadjrY5e+H4GIn9so\n FyCsBxAQk9OWPVGlowlBpi0uNHKJ7fL2PfILMso0j2TO1VXhs0DYMKD3rll0LfEPvpVVutmh0nz\n D7zA9AVKWIkpSg=","X-Received":["by 2002:a05:7300:2327:b0:2d8:97d6:6ac8 with SMTP id\n 5a478bee46e88-2e478c1ff0cmr3587224eec.21.1776493785549;\n Fri, 17 Apr 2026 23:29:45 -0700 (PDT)","by 2002:a05:7300:2327:b0:2d8:97d6:6ac8 with SMTP id\n 5a478bee46e88-2e478c1ff0cmr3587212eec.21.1776493784967; Fri, 17 Apr 2026\n 23:29:44 -0700 (PDT)"],"MIME-Version":"1.0","References":"","In-Reply-To":"","From":"Andrew Pinski ","Date":"Fri, 17 Apr 2026 23:29:34 -0700","X-Gm-Features":"AQROBzC_gzUgQ-GUGYtQn25JrR-GWB_xlUjqMuwKiqi9AYcBsV1clbLnBVcKGFE","Message-ID":"\n ","Subject":"Re: [PATCH] testsuite: flag failure to detect failures","To":"Alexandre Oliva ","Cc":"gcc-patches@gcc.gnu.org, Rainer Orth ,\n Mike Stump ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-Proofpoint-GUID":"blfylGU7JGNpF5UsUqY-tT36tUFuXac_","X-Authority-Analysis":"v=2.4 cv=KdDidwYD c=1 sm=1 tr=0 ts=69e324da cx=c_pps\n a=Uww141gWH0fZj/3QKPojxA==:117 a=IkcTkHD0fZMA:10 a=A5OVakUREuEA:10\n a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22\n a=_K5XuSEh1TEqbUxoQ0s3:22 a=1GBBUdQHAAAA:8 a=A8oJy8mYAAAA:8\n a=hWHKUdcDNR1jDJ5u8v0A:9 a=QEXdDO2ut3YA:10 a=PxkB5W3o20Ba91AHUih5:22\n a=PzjuB2m5y0pwU9Aq0M6F:22 a=MasS0_dV9q-jaWZIey9J:22","X-Proofpoint-ORIG-GUID":"blfylGU7JGNpF5UsUqY-tT36tUFuXac_","X-Proofpoint-Spam-Details-Enc":"AW1haW4tMjYwNDE4MDA1NyBTYWx0ZWRfXz+YMtv1LhuNJ\n LHSNHnB3DpuZwZsGd+HUtBxRCAyjA++K5IEK2RYtXv8BNOYyQaeC36mE6ukaJkkEw4tQ2D1qao0\n S78xH8dR5F5zYBx0E0MkMqSUXRPKKHrocXjqZ8eu2YCbOPJ3iW+Jq/9ylozMWC/TpON6oOpAxxW\n 85wEhiMQdDPj/BK7JuHUTDsJv8rvMUEU8EmelcsuAytfXiUcJbVh7O6KswryXbbuS497GEgxitF\n DHNeyopo5Kxdlo7rbew/a9IZ8BfzduQ/CSVxuwcvVDGAhnnBZ0tos7QN2MnU2t8wlrJmqxlB+gM\n tqkUJql+UyA2iw1bZoPVoZrLOIjq7TOY5mLTyup2GPgh6DlCXIqvLHvBJkirdzyH/YxaaETyP0h\n I0QrXxZEFGLlTQ/JretiHCcpF0hevroRQ2ven59bTlh3ryDUyKBpC6Pam7gguQx/fe1BeiTxdEJ\n wq+aKwKDNjr6R3ZgDQA==","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49\n definitions=2026-04-18_02,2026-04-17_04,2025-10-01_01","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n clxscore=1015 impostorscore=0 spamscore=0 priorityscore=1501 malwarescore=0\n suspectscore=0 adultscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0\n classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0\n reason=mlx scancount=1 engine=8.22.0-2604070000 definitions=main-2604180057","X-BeenThere":"gcc-patches@gcc.gnu.org","X-Mailman-Version":"2.1.30","Precedence":"list","List-Id":"Gcc-patches mailing list ","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org"}}]