[{"id":3676261,"web_url":"http://patchwork.ozlabs.org/comment/3676261/","msgid":"","list_archive_url":null,"date":"2026-04-12T11:20:54","subject":"Re: [PATCH 00/12] test: Add support for passing arguments to C unit\n tests","submitter":{"id":6170,"url":"http://patchwork.ozlabs.org/api/people/6170/","name":"Simon Glass","email":"sjg@chromium.org"},"content":"Hi,\n\nOn Sat, 11 Apr 2026 at 19:34, Simon Glass wrote:\n>\n> Some tests have two parts: a Python test which handles setup, such as\n> creating filesystem images or configuring the environment, and a C\n> test which does the bulk of the work including assertions. At present\n> there is no way to pass information between these two parts, so\n> things like paths and device names must be hard-coded in the C test.\n>\n> This tends to make people write tests in Python, but this is not ideal\n> for several reasons:\n>\n> - Python lacks access to things like return codes and has to scan the\n> console output to figure out what happened\n> - Python lacks access to internal state, so it cannot directly check\n> the result of an operation\n> - Python is ~100x slower due to console interaction, etc.\n> - C tests can be stepped through in gdb, which is much harder with\n> Python driving the console\n> - C tests can exercise internal APIs directly (e.g. fs_read(),\n> fs_size()) rather than going through the command layer, so they\n> test more precisely what they intend to\n>\n> This series adds infrastructure for passing typed, runtime arguments\n> from Python to C unit tests, using a name=value format on the ut\n> command line.\n>\n> The series includes:\n>\n> - Regex assertion helpers for matching variable output in tests\n>\n> - A private buffer in unit_test_state for test-local temporary data\n>\n> - Argument-type definitions (string, integer, boolean) with optional\n> flags and default values\n>\n> - A new UNIT_TEST_ARGS() macro for declaring tests with typed\n> arguments\n>\n> - Argument parsing in the test framework, accepting name=value format\n>\n> - Updates to the ut command to pass arguments through to tests\n>\n> - Type-checked accessor macros ut_str(), ut_int(), and ut_bool()\n> with bounds validation\n>\n> - Tests for the argument feature covering type checking, optional\n> arguments, and argument-parsing failures\n>\n> - Documentation for the test-parameter feature\n>\n> - C-based filesystem tests as an example of the hybrid approach,\n> with Python wrappers that pass filesystem type, image path, and\n> expected MD5 values to the C tests\n>\n>\n> Simon Glass (12):\n> test: Add ut_asserteq_regex() for regex pattern matching\n> test: Add a helper to check the next line against a regex\n> test: Add a private buffer for tests\n> test: Add argument-type definitions\n> test: Add a macro to declare unit tests with arguments\n> test: Add support for passing arguments to C tests\n> test: Enhance the ut command to pass test arguments\n> test: Add type-checked argument accessor functions\n> test: Add tests for unit-test arguments\n> test: Add documentation for the test framework\n> test: fs: add C-based filesystem tests\n> test: fs: Update Python tests to call C implementations\n>\n> arch/sandbox/cpu/spl.c | 3 +-\n> doc/develop/tests_writing.rst | 169 ++++++++++++\n> doc/usage/cmd/ut.rst | 18 +-\n> include/test/fs.h | 39 +++\n> include/test/test.h | 107 ++++++++\n> include/test/ut.h | 123 ++++++++-\n> test/Makefile | 1 +\n> test/cmd_ut.c | 32 ++-\n> test/common/Makefile | 1 +\n> test/common/test_args.c | 176 ++++++++++++\n> test/fs/Makefile | 3 +\n> test/fs/fs_basic.c | 407 ++++++++++++++++++++++++++++\n> test/py/tests/test_fs/conftest.py | 4 +-\n> test/py/tests/test_fs/test_basic.py | 346 +++++++----------------\n> test/test-main.c | 170 +++++++++++-\n> test/ut.c | 117 ++++++++\n> 16 files changed, 1437 insertions(+), 279 deletions(-)\n> create mode 100644 include/test/fs.h\n> create mode 100644 test/common/test_args.c\n> create mode 100644 test/fs/Makefile\n> create mode 100644 test/fs/fs_basic.c\n>\n> --\n> 2.43.0\n>\n> base-commit: e2fa3e570f83ab0f9ce667ddaec9dc738bcf05b9\n> branch: testa-us\n\nUnfortunately this one has double-signoff, so I will send a v2 with\nthis corrected.\n\nRegards,\nSimon","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256\n header.s=google header.b=AXgZGzrn;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de\n (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de;\n envelope-from=u-boot-bounces@lists.denx.de; receiver=patchwork.ozlabs.org)","phobos.denx.de;\n dmarc=pass (p=none dis=none) header.from=chromium.org","phobos.denx.de;\n spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de","phobos.denx.de;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=chromium.org header.i=@chromium.org\n header.b=\"AXgZGzrn\";\n\tdkim-atps=neutral","phobos.denx.de;\n dmarc=pass (p=none dis=none) header.from=chromium.org","phobos.denx.de;\n spf=pass smtp.mailfrom=sjg@chromium.org"],"Received":["from phobos.denx.de (phobos.denx.de\n [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4ftp5F0RXdz1xtJ\n\tfor ; Sun, 12 Apr 2026 21:21:20 +1000 (AEST)","from h2850616.stratoserver.net (localhost [IPv6:::1])\n\tby phobos.denx.de (Postfix) with ESMTP id 3A37B839D5;\n\tSun, 12 Apr 2026 13:21:11 +0200 (CEST)","by phobos.denx.de (Postfix, from userid 109)\n id 60B3183B99; Sun, 12 Apr 2026 13:21:10 +0200 (CEST)","from mail-ej1-x629.google.com (mail-ej1-x629.google.com\n [IPv6:2a00:1450:4864:20::629])\n (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits))\n (No client certificate requested)\n by phobos.denx.de (Postfix) with ESMTPS id C3A5A8341A\n for ; Sun, 12 Apr 2026 13:21:07 +0200 (CEST)","by mail-ej1-x629.google.com with SMTP id\n a640c23a62f3a-b932fe2e1a7so474386766b.1\n for ; Sun, 12 Apr 2026 04:21:07 -0700 (PDT)"],"X-Spam-Checker-Version":"SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de","X-Spam-Level":"","X-Spam-Status":"No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,\n DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,\n RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham\n autolearn_force=no version=3.4.2","ARC-Seal":"i=1; a=rsa-sha256; t=1775992866; cv=none;\n d=google.com; s=arc-20240605;\n b=TD92TWNTHHxrCyRO/Ybc4MsBhJe79FQ1Femc5MYsu7hzIaD2YLXvKZPmf+5+SoNMwX\n sAiBPt+oO7QPOUh5ejfSS86/rvbmpFLkY1FtbbibeaopjyGqeF6TArCCBMo9u76yHvSG\n J9of0bHzMmzYTsyS+BL91OlNWpBx1W05WTEiRJCekn7pOc1Yd3bCwX0fMfGWFE2OrGFA\n WhvCyCALkko2GTpzC6UF6GtkBvmVsu6H6LKFtybVj0s+1V3vte1hXQNtxirizfUBnzvb\n WQTOw1zf53iZunEyxHXPzvcy/TbQzfgug1tLXRjtJihWYCGQBayUq1wak2ILZJjkZtCL\n jPvg==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;\n s=arc-20240605;\n h=cc:to:subject:message-id:date:from:in-reply-to:references\n :mime-version:dkim-signature;\n bh=3wC5G5vHVRfBRy/DxkMrvdi41uaeXKHwicDiRkQAKCI=;\n fh=bTV0IwHclDB8+vGfvycXQoaDQhlmAUt+vuhaiICa/Ug=;\n b=URweCplEOrzTvrh1Cd8C7Y2HOeAg2lU0EIXmHM7G+pivPdQwIjz4yvY2KnqChPLYE+\n CRRSpvlD6H3stxy8kRkhIAp06PUdr9TC2ceC00ZvEkYY7wiUoRj5r+/2gfFDipl62wo7\n 53wyGLLoI/OD+0cKwdsxiUlmr+OBpTrUc4OoUR5IKlK1jQlBLFd55QGk7quWPraWHRJ/\n ygbPrjydLdC70h+OdKb40m1GwogA2MIBKfpzAU6laDeddhXwu2nvIw5TSqDWz5IBXCXy\n 5RiHiAW+DVbFvEtwvPdudwzQ7lrvwPfVuqE9b5pJPuIJocq49X8WU58MGbo1YYJd3SDu\n v8ag==; darn=lists.denx.de","ARC-Authentication-Results":"i=1; mx.google.com; arc=none","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=chromium.org; s=google; t=1775992866; x=1776597666; darn=lists.denx.de;\n h=cc:to:subject:message-id:date:from:in-reply-to:references\n :mime-version:from:to:cc:subject:date:message-id:reply-to;\n bh=3wC5G5vHVRfBRy/DxkMrvdi41uaeXKHwicDiRkQAKCI=;\n b=AXgZGzrnyaAmoGcrxKzQcX4NURysQEahL3KTKwHz8am+cTfmbcX9/47LE7d2nirPUj\n qfzlbAaWED61H20VoXkEmI4pYxZ6y+HrNi26rgK7RLGeEdJoaWq43m5a2nAQTugLS/Xt\n nqBOYK5oMjrKcV19UEpOjCWPaZ7tqN6bsiRIU=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20251104; t=1775992866; x=1776597666;\n h=cc:to:subject:message-id:date:from:in-reply-to:references\n :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=3wC5G5vHVRfBRy/DxkMrvdi41uaeXKHwicDiRkQAKCI=;\n b=VXFFNdkAOO7NH1T8ctDeXhF9qFdGAjbalTL5s07G2rbsliuhvx0dSpQpksO0zTF05X\n GjXj4XHAOun1HzEMV03rdyCHwa0eqyzRJDKXl8IdEWKzPuSRxEz/b25xhrf+WqiEf3BL\n NS7klNdIKQF28pBVkNPNPf6WHJzjkII7Htkw+HPXBr+niCnXy+B8PKEl3AllMqzYS9bh\n YF5EsWrvmszaGRRkJGE9QU2epKAvx4CJ1mzmQAy15nzwFU1j+Pd1kIa4LFE0GxyzNnPN\n VWNuMbJg0/r07IbS2k59PkG0QIEeHfVy+t5KfOgTJUSQFQo5RRK3t4xVUgjw2CpFY6Db\n 7DsQ==","X-Gm-Message-State":"AOJu0YzecBO4+NOPi+Yyv5Qo9uY8KEzUo2/h3Zw9xvn0igJlcL0WQQ1c\n erlUY031gme2ID58r1iaBA4mNI0NzCMlPXfo3BULHndQZfiWLzOjqE/i+uUHwTsc8dsXT4OT4td\n jCQhx4OntqZ6usXtUiJU98OJu4GwgmcXdaaqKKFK3r6VqXaiVCNY=","X-Gm-Gg":"AeBDieuXl+g07lTquTt78zDGQV9zf9yJCfiI+nP4AuoW7S92ICOK7hwj7XAw60ac1VK\n xMMytqmuhJxBTW6Bvsdsw5pm0V+Hbtom9OZVHzlZyYefjcPaGyUbKSWrHIESc6QyKP/2IjU7pC/\n +r13nKHDRh82eeSj4R4wU6oyQ9paDi79f2luXnWobpKhZ4ok5wBqFtVEEBdlbNSTVjS1o87V8hU\n A0Ff3TCcxj4D2Z62Z4rdrtrCfaut9fLnm/HXZpDgXgNL8quAISVm6oYI9LF/NHFxMTpsnwCwLvW\n pEC49w==","X-Received":"by 2002:a17:907:3f87:b0:b9d:df73:f0b8 with SMTP id\n a640c23a62f3a-b9ddf7429e8mr24655166b.32.1775992866044; Sun, 12 Apr 2026\n 04:21:06 -0700 (PDT)","MIME-Version":"1.0","References":"<20260412013451.2929001-1-sjg@chromium.org>","In-Reply-To":"<20260412013451.2929001-1-sjg@chromium.org>","From":"Simon Glass ","Date":"Sun, 12 Apr 2026 05:20:54 -0600","X-Gm-Features":"AQROBzDnLA5OkDPS9VIqcabCUlNK59ck0p88FM7YnHd9kqHg-Vbn-649125Gl7o","Message-ID":"\n ","Subject":"Re: [PATCH 00/12] test: Add support for passing arguments to C unit\n tests","To":"u-boot@lists.denx.de","Cc":"Heinrich Schuchardt , Tom Rini ,\n Andrew Goodbody ,\n Ilias Apalodimas ,\n Marek Vasut ","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"u-boot@lists.denx.de","X-Mailman-Version":"2.1.39","Precedence":"list","List-Id":"U-Boot discussion ","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"u-boot-bounces@lists.denx.de","Sender":"\"U-Boot\" ","X-Virus-Scanned":"clamav-milter 0.103.8 at phobos.denx.de","X-Virus-Status":"Clean"}}]