Cover Letter Detail
Show a cover letter.
GET /api/covers/2215747/?format=api
{ "id": 2215747, "url": "http://patchwork.ozlabs.org/api/covers/2215747/?format=api", "web_url": "http://patchwork.ozlabs.org/project/linux-ext4/cover/20260325093349.630193-1-diangangli@gmail.com/", "project": { "id": 8, "url": "http://patchwork.ozlabs.org/api/projects/8/?format=api", "name": "Linux ext4 filesystem development", "link_name": "linux-ext4", "list_id": "linux-ext4.vger.kernel.org", "list_email": "linux-ext4@vger.kernel.org", "web_url": null, "scm_url": null, "webscm_url": null, "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<20260325093349.630193-1-diangangli@gmail.com>", "list_archive_url": null, "date": "2026-03-25T09:33:48", "name": "[RFC,0/1] ext4: fail fast on repeated metadata reads after IO failure", "submitter": { "id": 92966, "url": "http://patchwork.ozlabs.org/api/people/92966/?format=api", "name": "Diangang Li", "email": "diangangli@gmail.com" }, "mbox": "http://patchwork.ozlabs.org/project/linux-ext4/cover/20260325093349.630193-1-diangangli@gmail.com/mbox/", "series": [ { "id": 497401, "url": "http://patchwork.ozlabs.org/api/series/497401/?format=api", "web_url": "http://patchwork.ozlabs.org/project/linux-ext4/list/?series=497401", "date": "2026-03-25T09:33:49", "name": "ext4: fail fast on repeated metadata reads after IO failure", "version": 1, "mbox": "http://patchwork.ozlabs.org/series/497401/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/covers/2215747/comments/", "headers": { "Return-Path": "\n <SRS0=WUEV=BZ=vger.kernel.org=linux-ext4+bounces-15358-patchwork-incoming=ozlabs.org@ozlabs.org>", "X-Original-To": [ "incoming@patchwork.ozlabs.org", "linux-ext4@vger.kernel.org" ], "Delivered-To": [ "patchwork-incoming@legolas.ozlabs.org", "patchwork-incoming@ozlabs.org" ], "Authentication-Results": [ "legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256\n header.s=20251104 header.b=a/asXWCV;\n\tdkim-atps=neutral", "legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=ozlabs.org\n (client-ip=150.107.74.76; helo=mail.ozlabs.org;\n envelope-from=srs0=wuev=bz=vger.kernel.org=linux-ext4+bounces-15358-patchwork-incoming=ozlabs.org@ozlabs.org;\n receiver=patchwork.ozlabs.org)", "gandalf.ozlabs.org;\n arc=pass smtp.remote-ip=172.234.253.10 arc.chain=subspace.kernel.org", "gandalf.ozlabs.org;\n dmarc=pass (p=none dis=none) header.from=gmail.com", "gandalf.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256\n header.s=20251104 header.b=a/asXWCV;\n\tdkim-atps=neutral", "gandalf.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org\n (client-ip=172.234.253.10; helo=sea.lore.kernel.org;\n envelope-from=linux-ext4+bounces-15358-patchwork-incoming=ozlabs.org@vger.kernel.org;\n receiver=ozlabs.org)", "smtp.subspace.kernel.org;\n\tdkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com\n header.b=\"a/asXWCV\"", "smtp.subspace.kernel.org;\n arc=none smtp.client-ip=209.85.214.174", "smtp.subspace.kernel.org;\n dmarc=pass (p=none dis=none) header.from=gmail.com", "smtp.subspace.kernel.org;\n spf=pass smtp.mailfrom=gmail.com" ], "Received": [ "from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519 server-signature ECDSA (secp384r1 raw public key)\n server-digest SHA384)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4fghkn6H9Rz1xy3\n\tfor <incoming@patchwork.ozlabs.org>; Wed, 25 Mar 2026 20:41:53 +1100 (AEDT)", "from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3])\n\tby gandalf.ozlabs.org (Postfix) with ESMTP id 4fghkn626nz4w11\n\tfor <incoming@patchwork.ozlabs.org>; Wed, 25 Mar 2026 20:41:53 +1100 (AEDT)", "by gandalf.ozlabs.org (Postfix)\n\tid 4fghkn5xMXz4wHj; Wed, 25 Mar 2026 20:41:53 +1100 (AEDT)", "from sea.lore.kernel.org (sea.lore.kernel.org [172.234.253.10])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519)\n\t(No client certificate requested)\n\tby gandalf.ozlabs.org (Postfix) with ESMTPS id 4fghkk3plqz4w11\n\tfor <patchwork-incoming@ozlabs.org>; Wed, 25 Mar 2026 20:41:50 +1100 (AEDT)", "from smtp.subspace.kernel.org (conduit.subspace.kernel.org\n [100.90.174.1])\n\tby sea.lore.kernel.org (Postfix) with ESMTP id D342E30DB0A0\n\tfor <patchwork-incoming@ozlabs.org>; Wed, 25 Mar 2026 09:34:34 +0000 (UTC)", "from localhost.localdomain (localhost.localdomain [127.0.0.1])\n\tby smtp.subspace.kernel.org (Postfix) with ESMTP id E0C5F359A7E;\n\tWed, 25 Mar 2026 09:34:33 +0000 (UTC)", "from mail-pl1-f174.google.com (mail-pl1-f174.google.com\n [209.85.214.174])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))\n\t(No client certificate requested)\n\tby smtp.subspace.kernel.org (Postfix) with ESMTPS id 71ADA30CD92\n\tfor <linux-ext4@vger.kernel.org>; Wed, 25 Mar 2026 09:34:31 +0000 (UTC)", "by mail-pl1-f174.google.com with SMTP id\n d9443c01a7336-2b0b0064027so6265455ad.3\n for <linux-ext4@vger.kernel.org>;\n Wed, 25 Mar 2026 02:34:31 -0700 (PDT)", "from n37-098-250.byted.org ([115.190.40.15])\n by smtp.gmail.com with ESMTPSA id\n d9443c01a7336-2b083516ae1sm164266245ad.13.2026.03.25.02.34.27\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Wed, 25 Mar 2026 02:34:30 -0700 (PDT)" ], "ARC-Seal": [ "i=2; a=rsa-sha256; d=ozlabs.org; s=201707; t=1774431713; cv=pass;\n\tb=Ko8x/DQn2R9IDPwJr7oquWQeyCYq5N7RjhoZizbmzmfOB5IKDdnQEL/R2LlbRcWq2C7iglx9E0NHjXalbsywhCeio75WrJsvT/fqzuUzs8rgxXXrG1S9xrrJRqaiACfWycpqNu9xLB6fM+VwLcaLqM4rAma2R5smCoe2DVkn4cH3rqBkODhwsz4TX46rS+ilPnzakHLs/cCckVZhfc4+G8i5Vhfbu+UMX5OiwsI9etVvcsaGhFD7knCkUycsS+H369+dmELjPUQKzo8HzgMrrJfMCfy4+bWNrP30BEOBihm1jbgnIbAUx0btO3p60LvEX67Ljet3RbKoUgBoah5x/A==", "i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;\n\tt=1774431273; cv=none;\n b=SYXLwVQJP2WlJ7hslEaE2+ap6vD7lWTJ45weWPHzeIUZveV0SFN0dJh48Ml9lkS37e1kMumnrmvakrBbg71A1Sve+/kvmCQEQhWo4PpnBfMmbNwob1X8venrHyYkejXQsO+uQT6utH/X6mbhPIIBgTyBr4oPMwn5QAqTwhIZHpo=" ], "ARC-Message-Signature": [ "i=2; a=rsa-sha256; d=ozlabs.org; s=201707;\n\tt=1774431713; c=relaxed/relaxed;\n\tbh=6bSkic//eLc54cP3e25oSENPT2tB4RkJc8+gyvzAmqU=;\n\th=From:To:Cc:Subject:Date:Message-Id:MIME-Version;\n b=yPbH5bgCcql6tAPWd7t8eQ88EnXUlAPkMok3+tSrFYPZw7v0mT31YbXDj5hUGMajXsSNBdfL4f2QTUrL8aq2pHNLTk/0talyMI+8q0F3Xgqhxf/zP2yiTcGxSownYjahId3HndFTUuFuuD/CEYyRnbsvFjbYb9xDMClT5KGFOzTCuxwMUQxA3RJZEWDkJDeu9dIPkY1E0XoG+L6GZzTqe8t2ZAmfBlgf6Q3dI7HyS/oI8Q5qEGDWxMqe9gnfbT6VZFLMDVbklF3OLdQD3YvgQZwGRBwqjEZojvqQlyJH+Ad+OxHkZice4q3t05xtYcQfxBgo/bmS9wP+jm5GNkEAmw==", "i=1; a=rsa-sha256; d=subspace.kernel.org;\n\ts=arc-20240116; t=1774431273; c=relaxed/simple;\n\tbh=zhfXxitK3GbVzP0cWaWb05D74FMylePtHteof0l3m0A=;\n\th=From:To:Cc:Subject:Date:Message-Id:MIME-Version;\n b=YacFOVV0KD3e6t8zY2ymK/RX7w6TjkKagd3mTrG+Ucw+M7W4cmcC3QRhveNWWxhU/PdM/AWRBGQ2Xoiq/lAm5X1Vlc3ottYWI2PiVlfZ5HJteju7+1KlUkL/s5otnmW30lyorRtcm8QtWRIFYBvM24ncrjMVqfl+3KlFvmbn/K8=" ], "ARC-Authentication-Results": [ "i=2; gandalf.ozlabs.org;\n dmarc=pass (p=none dis=none) header.from=gmail.com; dkim=pass (2048-bit key;\n unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256\n header.s=20251104 header.b=a/asXWCV; dkim-atps=neutral;\n spf=pass (client-ip=172.234.253.10; helo=sea.lore.kernel.org;\n envelope-from=linux-ext4+bounces-15358-patchwork-incoming=ozlabs.org@vger.kernel.org;\n receiver=ozlabs.org) smtp.mailfrom=vger.kernel.org", "i=1; smtp.subspace.kernel.org;\n dmarc=pass (p=none dis=none) header.from=gmail.com;\n spf=pass smtp.mailfrom=gmail.com;\n dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com\n header.b=a/asXWCV; arc=none smtp.client-ip=209.85.214.174" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=gmail.com; s=20251104; t=1774431271; x=1775036071;\n darn=vger.kernel.org;\n h=content-transfer-encoding:mime-version:message-id:date:subject:cc\n :to:from:from:to:cc:subject:date:message-id:reply-to;\n bh=6bSkic//eLc54cP3e25oSENPT2tB4RkJc8+gyvzAmqU=;\n b=a/asXWCVpW864kGkkSn/0Z8Uf1Qyw1Wr+MBvsCpYCuVPj1604CKshxlMPzhBWRm5Ep\n AbN/jm0+Ht4QElFhNuKaYBLdTXtfJcvYS0sGfmq/u5962dUTO15dBKkXtbYpLXFNJOK0\n ZRjz9rAD27slX+ZqmkmmePk4w6K3TttssAz+oc9FIsPmNRNWDc71Upq4ADbom0M01g+k\n +5fTKtIy0rq88YLLBMy0GPTqws2fnOkB9iP4JuYvKcQDq+eBMsNOFnQ8NMP8NQNkpjhS\n k1H6g3NQgv2TOsN28vMkh5k3Ekt3Ym+GTHeNJiOm6bvjWTiN7Q5F38w81laRuT0tcbhC\n 4zuA==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20251104; t=1774431271; x=1775036071;\n h=content-transfer-encoding:mime-version:message-id:date:subject:cc\n :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=6bSkic//eLc54cP3e25oSENPT2tB4RkJc8+gyvzAmqU=;\n b=qBGTBTCchanl+ZWRR9cUdm43A1Kg/hy/THwOWDynKcuSx1vsoVmkmz87XXmjrYYvNy\n 8dvQIzOQLvCN5f9N1g83ebwxk1KjnAouj/sHE+Zh1oyqOjxJ5VRohH1/JuRmFfa18eDf\n +9lhJDjn8H8ZdaDF2H5Ta2c9cavJW6OaDtQlPORrJ2jzzAT7VVQ7htxzUWpQykYI/hfN\n Br5dhP2fprUX7AdRPHu0ESkrpB7rd+o3T0u1OQlV9x1+RXAl+Eik6427kU/+931R/yS1\n GzZJoTOWi2mXTPeXZ0tbTfgswer93KAqZzr4IH+dtjlfgZc3+DY7sMS6MfOsOrTs4Naq\n krbQ==", "X-Gm-Message-State": "AOJu0Yz7FK1vdMsSZ5zNp+ECFLsIlLo6YlDSJqO4/qoKyaMPs6MeODmB\n\trqBeSItpFmv8MgG8UzUnn45RmoT5ZCEx2fk5iAP1vNpaKCAR0Ca8Cbfm", "X-Gm-Gg": "ATEYQzzLR0gAZQCpwUk/yyeN9K1FK7Qk7vfoRkhYnE2aR4W8Ux6bLHyrqSEO6xoGrlS\n\tsXlno+jHESz1cuUlXYO/JrPuZ+g0oZ3aSuBjbiia23t3bGhSLfWC3MbInJ3vhZdS7Z8TP+eV7gY\n\tIx1k/agYuFmpkRtPDufsZ9NZrqM1heXeaz1AQqqQeiriuVAixUSw+4Bklfl7Lt7VfLjIKxSUlKz\n\tD/5AVWB6/OidyFbi5ekkJfdMlR3ROCnQm82HnuoRCN8h6KtDhTuSQG3XrlaCTQFrZhyiq69OwB/\n\tam7V1uTdj3H9o8RSZXhPy2juqKlw5dNt2x7Qj1nXc5mU3wO4d+nbDHGaZFPn+m9sf/9YpplpVfE\n\tYzZdbFRHupjd/cmmwY+dAJIb8gNCtiPgchbwmssQvV/YygcffVqhFvexsBTvR0L+gM3XGSh6vp/\n\t+yjEHBvjMzhVIu3apZQArbL852mAjrd9y2LToD3FScrRTcI/4=", "X-Received": "by 2002:a17:903:1786:b0:2b0:5770:d484 with SMTP id\n d9443c01a7336-2b0b0ade793mr29220765ad.41.1774431270540;\n Wed, 25 Mar 2026 02:34:30 -0700 (PDT)", "From": "Diangang Li <diangangli@gmail.com>", "To": "tytso@mit.edu,\n\tadilger.kernel@dilger.ca", "Cc": "linux-ext4@vger.kernel.org,\n\tlinux-fsdevel@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org,\n\tchangfengnan@bytedance.com,\n\tDiangang Li <lidiangang@bytedance.com>", "Subject": "[RFC PATCH 0/1] ext4: fail fast on repeated metadata reads after IO\n failure", "Date": "Wed, 25 Mar 2026 17:33:48 +0800", "Message-Id": "<20260325093349.630193-1-diangangli@gmail.com>", "X-Mailer": "git-send-email 2.39.5", "Precedence": "bulk", "X-Mailing-List": "linux-ext4@vger.kernel.org", "List-Id": "<linux-ext4.vger.kernel.org>", "List-Subscribe": "<mailto:linux-ext4+subscribe@vger.kernel.org>", "List-Unsubscribe": "<mailto:linux-ext4+unsubscribe@vger.kernel.org>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-Spam-Status": "No, score=-1.2 required=5.0 tests=ARC_SIGNED,ARC_VALID,\n\tDKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DMARC_PASS,\n\tFREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,\n\tMAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=disabled\n\tversion=4.0.1", "X-Spam-Checker-Version": "SpamAssassin 4.0.1 (2024-03-25) on gandalf.ozlabs.org" }, "content": "From: Diangang Li <lidiangang@bytedance.com>\n\nA production system reported hung tasks blocked for 300s+ in ext4 metadata\nlookup paths. Hung task reports were accompanied by disk IO errors, but\nprofiling showed that most individual reads completed (or failed) within\n10s, with the worst case around 60s.\n\nAt the same time, we observed a high repeat rate to the same disk LBAs.\nThe repeated reads frequently showed seconds-level latency and ended with\nIO errors, e.g.:\n\n [Tue Mar 24 14:16:24 2026] blk_update_request: I/O error, dev sdi,\n sector 10704150288 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0\n [Tue Mar 24 14:16:25 2026] blk_update_request: I/O error, dev sdi,\n sector 10704488160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0\n [Tue Mar 24 14:16:26 2026] blk_update_request: I/O error, dev sdi,\n sector 10704382912 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0\n\nWe also sampled repeated-LBA latency histograms on /dev/sdi and saw that\nthe same error-prone LBAs were re-submitted many times with ~1-4s latency:\n\n LBA 10704488160 (count=22): 1-2s: 20, 2-4s: 2\n LBA 10704382912 (count=21): 1-2s: 20, 2-4s: 1\n LBA 10704150288 (count=21): 1-2s: 19, 2-4s: 2\n\nRoot cause\n==========\n\next4 metadata reads commonly use buffer_head caching and serialize IO via\nBH_Lock (lock_buffer). When a read fails, the buffer remains !Uptodate.\nWith multiple threads concurrently accessing the same metadata block, each\nwaiter wakes up after the previous owner drops BH_Lock, then submits the\nsame read again and waits again. This makes the latency grow linearly with\nthe number of contending threads, leading to 300s+ hung tasks.\n\nThe failing IOs are repeatedly issued to the same LBA. The observed 1s+\nper-IO latency is likely from device-side retry/error recovery. On SCSI the\ndriver typically retries reads several times (e.g. 5 retries in our\nenvironment), so a single filesystem submission can easily accumulate 5s+\ndelay before failing. When multiple threads then re-submit the same failing\nmetadata read and serialize on BH_Lock, the delay is amplified into 300s+\nhung tasks.\n\nSimilar behavior exists for other devices (e.g. NVMe with multiple internal\nretries).\n\nExample hung stacks:\n\n INFO: task toutiao.infra.t:3760933 blocked for more than 327 seconds.\n Call Trace:\n __schedule\n io_schedule\n __wait_on_bit_lock\n bh_uptodate_or_lock\n __read_extent_tree_block\n ext4_find_extent\n ext4_ext_map_blocks\n ext4_map_blocks\n ext4_getblk\n ext4_bread\n __ext4_read_dirblock\n dx_probe\n ext4_htree_fill_tree\n ext4_readdir\n iterate_dir\n ksys_getdents64\n\n INFO: task toutiao.infra.t:2724456 blocked for more than 327 seconds.\n Call Trace:\n __schedule\n io_schedule\n __wait_on_bit_lock\n ext4_read_bh_lock\n ext4_bread\n __ext4_read_dirblock\n htree_dirblock_to_tree\n ext4_htree_fill_tree\n ext4_readdir\n iterate_dir\n ksys_getdents64\n\nApproach\n========\n\nRemember read failures on buffer_head and fail fast for ext4 metadata reads\nonce a buffer has already seen a read failure. Clear the flag on successful\nread/write completion so the buffer can recover if the error is transient.\n\nNote that ext4 read-ahead uses ext4_read_bh_nowait(), so it does not set\nthe failure flag and remains best-effort.\n\nPatch summary\n=============\n\n 1) Add BH_Read_EIO buffer_head state bit and helpers.\n 2) Clear BH_Read_EIO on successful read/write completion.\n 3) In ext4 metadata reads, if BH_Read_EIO is already set (and not\n BH_Write_EIO), fail fast instead of re-submitting the same failing\n read. On read failure, set BH_Read_EIO.\n\nDiangang Li (1):\n ext4: fail fast on repeated metadata reads after IO failure\n\n fs/buffer.c | 2 ++\n fs/ext4/super.c | 12 +++++++++++-\n include/linux/buffer_head.h | 2 ++\n 3 files changed, 15 insertions(+), 1 deletion(-)" }