From patchwork Tue Nov 7 18:07:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861204 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=F62o9/q2; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPx8w16Dvz1yQL for ; Wed, 8 Nov 2023 05:10:40 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QX1-0003WO-UN; Tue, 07 Nov 2023 13:10:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QX0-0003UM-BV for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:30 -0500 Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWr-0002ix-2P for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:30 -0500 Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-5aa481d53e5so4080429a12.1 for ; Tue, 07 Nov 2023 10:10:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380617; x=1699985417; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ogYJbqlkzS2Yd8saws9Occt0F7m6RWm2l0bkCCgcR3U=; b=F62o9/q2Mqu8GS0dfhOZyso7RT6hASuHOCoSBfoWochdmUNnR43aULGM+0aEMMVaZP wVd4+lz3sLFyilY67L46o3+lYvnDCV4Sc5Et5sR/8hniMvQsup2jMGkkqPNKOOt2P5ni GGeZunmaCbYrtmbI3pp5BXqHrSbVtbHqNIA8ymR34U1SQ9tCT6rk4OMZC7Pz6/Xz2k89 II/kekCTdAE8QCwcYX+mJgCgV84ZcTov7eaEM7EqUM/Sh387MPpisSEvQRmpFbg+bsg0 nkK1GMx2+QbSMxxHO99ujFur4uY7FPIu0kFQOIDkmgtr0wbX7xnK1yyJgvZPBxJb1FCZ p5Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380617; x=1699985417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ogYJbqlkzS2Yd8saws9Occt0F7m6RWm2l0bkCCgcR3U=; b=BAz1oXcxEHD5gS81mkTkOEeA/MeZQpZMja/7oaAnVCn/kUSDxD+KM/KeSG4DSMjyKF hyAksOwA4WqWczyxsN0fZxhUwe7yTcQyJWRxy0ccnqmLTvOrI+xnsAyiiPmRQgI9CBuS P256TvsOMUsMDATUTIv5sL+ioWilO7qlOMQDuXbffLbcRJafPL3bBP+I95azxbvQN8Y/ wcB17Ny+PMNSLiY0puwxu0UZua1wI+Ww9icqULAQDosJGh+c9yuVqp8MxPlLVliFlh7C bllIqtYm58UwGiIi2X61/CwOGRn4vZbX4YnJMi39vktiw5K0neILkHewNhBWAEpaePPo WgoQ== X-Gm-Message-State: AOJu0Yz+hqxJpG6/pW9gJE/liyz9oWHgwotsEW1l73X8biRyUKOV/11N bH2Wrb6FGlokIPv/3WZWU/ArrWJx6B8= X-Google-Smtp-Source: AGHT+IFWm9kgjVGfWor3OqA8aZVUBtUhbkrx5BtmA6zSu7sqrwDUHquMPZD7UdUq5z+wxXKrUcUVUg== X-Received: by 2002:a17:90b:4acb:b0:280:3650:382a with SMTP id mh11-20020a17090b4acb00b002803650382amr22482838pjb.16.1699380617158; Tue, 07 Nov 2023 10:10:17 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:16 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 1/9] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Tue, 7 Nov 2023 10:07:05 -0800 Message-ID: <20231107180907.553451-2-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::531; envelope-from=nifan.cxl@gmail.com; helo=mail-pg1-x531.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Based on CXL spec 3.0 Table 8-94 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index f1145e9671..8eceedfa87 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -753,8 +754,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -780,6 +782,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Tue Nov 7 18:07:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861203 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=jpo27UHU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPx8n51fpz1yQL for ; Wed, 8 Nov 2023 05:10:33 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QWu-0003Lz-KI; Tue, 07 Nov 2023 13:10:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QWs-0003KE-49 for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:22 -0500 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWq-0002jA-1J for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:21 -0500 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1cc2f17ab26so44270085ad.0 for ; Tue, 07 Nov 2023 10:10:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380618; x=1699985418; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pxa2X/M3i5M/B67pIBM08iOpkqVk5AqzieyHZGjyhtc=; b=jpo27UHUwQ5GljXwIkqPvDrMrDlsAlR6LUPRrLR07bzy+NoTRDbSLQAdJGwHrmIx61 gURDSo2uW1QjeeWeMnwRywnPHdWbyXjHqz7ijcEBXobxdsOEYQYQyEJZV31SrrcJomLm JQQ7+uWUsRYWyTi+S+haFO37hAi+++Hw9CQmhfpxH6A9AWw09A+t9M9uJeeBhQapcxg6 n0K6FvCyBakHietp9+cbvqwTQ4L6/zEF4uVieIokEhbktXTMw2E6yo37GdOfqdRNcCPQ hPhm06bm7tQWr2/qU6cDTsMWeVBSdVX41OBPzWXMLoVDLOv/l9lZBG4DfFr+ceuFTyw9 gsEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380618; x=1699985418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pxa2X/M3i5M/B67pIBM08iOpkqVk5AqzieyHZGjyhtc=; b=KFgIhNgutoAbWd8MpHHA3KB/hle8i6ikZvpB9j9iBorzf1EXzuQW0dQUKnYkHTo50N ioZ+rCTh0q6cOzp0L2VGY7880q6RhnP4Kf4p77jTELSd77crD1KoSh64hWI/WnCEOXz2 H1XmNo48lM6Ac52HE+kRM7PDm7byw512mLQbmCXLpSYAT0K1M4VH5uS0Yn+ehQUO45BI fSrBgQI6KTKXeJIaQgwgKhPXXWSOo9+n8VmWpoBkPHdQuuniygBto6L8ewJz76wgztW7 YNqzSQ6iX2hMwSsUXmQk1dtTHOizOEokakxfRs06HDYf37R3vSjxNTebMTZc2awGON7E 5hcA== X-Gm-Message-State: AOJu0Yxz3OjtsECP5ka478PuQQLGG5ZVa34QUqJVGonGRy251w1Y0/kn SDbdQNnDC0GVQZYrN3b3TW57Hqq1DOE= X-Google-Smtp-Source: AGHT+IEQmjibQ784vy6VRNnHQ85xLgjNWXzyE4NL1vQ8Kkz/qut4AdhAt5wqtkfkCdPsz3SW1ZwoXg== X-Received: by 2002:a17:902:fa07:b0:1cb:ffc1:23a3 with SMTP id la7-20020a170902fa0700b001cbffc123a3mr26987897plb.62.1699380618084; Tue, 07 Nov 2023 10:10:18 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:17 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 2/9] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Tue, 7 Nov 2023 10:07:06 -0800 Message-ID: <20231107180907.553451-3-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Per cxl spec 3.0, add dynamic capacity region representative based on Table 8-126 and extend the cxl type3 device definition to include dc region information. Also, based on info in 8.2.9.8.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: decode_len of a dc region is aligned to 256*MiB, need to be divided by 256 * MiB before returned to the host for "Get Dynamic Capacity Configuration" mailbox command. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 80 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 6 +++ include/hw/cxl/cxl_device.h | 17 ++++++++ 3 files changed, 103 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 8eceedfa87..f80dd6474f 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -80,6 +80,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1210,6 +1212,74 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.0 section 8.2.9.8.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct get_dyn_cap_config_in_pl { + uint8_t region_cnt; + uint8_t start_region_id; + } QEMU_PACKED; + + struct get_dyn_cap_config_out_pl { + uint8_t num_regions; + uint8_t rsvd1[7]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + } QEMU_PACKED; + + struct get_dyn_cap_config_in_pl *in = (void *)payload_in; + struct get_dyn_cap_config_out_pl *out = (void *)payload_out; + uint16_t record_count = 0, i; + uint16_t out_pl_len; + uint8_t start_region_id = in->start_region_id; + + if (start_region_id >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_region_id, + in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + memset(out, 0, out_pl_len); + out->num_regions = record_count; + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_region_id + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_region_id + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_region_id + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_region_id + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_region_id + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_region_id + i].flags; + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1254,6 +1324,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 18 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1465,7 +1540,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 7b4d1ee774..6c1ccda159 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1075,6 +1075,12 @@ static void ct3d_reset(DeviceState *dev) uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; uint32_t *write_msk = ct3d->cxl_cstate.crb.cache_mem_regs_write_mask; + if (ct3d->dc.num_regions) { + ct3d->cxl_dstate.is_dcd = true; + } else { + ct3d->cxl_dstate.is_dcd = false; + } + cxl_component_register_init_common(reg_state, write_msk, CXL2_TYPE3_DEVICE); cxl_device_register_init_t3(ct3d); diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 4f2ef0b899..334c51fddb 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -235,6 +235,7 @@ typedef struct cxl_device_state { uint64_t mem_size; uint64_t pmem_size; uint64_t vmem_size; + bool is_dcd; const struct cxl_cmd (*cxl_cmd_set)[256]; CPMUState cpmu[CXL_NUM_CPMU_INSTANCES]; @@ -417,6 +418,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_REGION_NUM 8 + +typedef struct CXLDCDRegion { + uint64_t base; + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCDRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -453,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCDRegion regions[DCD_MAX_REGION_NUM]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Tue Nov 7 18:07:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861208 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Hg7oBFpF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPxBv2qdVz1yQL for ; Wed, 8 Nov 2023 05:12:23 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXC-0004Y1-Tm; Tue, 07 Nov 2023 13:10:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QX7-0004GR-Nm for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:37 -0500 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWt-0002jX-KN for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:37 -0500 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1cc9784dbc1so36416225ad.2 for ; Tue, 07 Nov 2023 10:10:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380619; x=1699985419; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lerGkvWs+2MRxdzE0DMBKiSOqV9ui2+nfnjyOq/W8aE=; b=Hg7oBFpFydTXSwDPBqQJ1H+8+J4Za4lOPO4UkYudRbeHUvvfbNlN0yCHxFf4AgeECV ChhUEuMLKo0rfMjy8bc8kfehEY7EETEX6Ovz1UJ2Cn5jRNqYxiBuX0UYYRFW/vNqMEQI ruhjpD61rjioPTt7fcB6wgNSipFi9IDcaquJLuni/b2i4Ah4NkzDcRYr6kQz6gcquodz 6J3VKUpTM5PaLN2pCKu6sNB4xAcbOhF9gLIQK9mW5UXpPH1wsveWm0k7y2nxRl8cqnYw 8SfvLdSNjN2z5qgEosu51lyQ3+qVibrDd+xk0eVLxEff/KgF3eVWbvlUmtRX8NO4+t+k BGaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380619; x=1699985419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lerGkvWs+2MRxdzE0DMBKiSOqV9ui2+nfnjyOq/W8aE=; b=FzFX8po+LYLsYbAN504qHTubb/WFeFuVbIJczwQLPIKnOIba5hHMYrvcBbuZiF2M+6 kHaf00sN17d2GcLIY/KU8+nLOKilx1fytBgVfleqGPFepuq0KHJ4T0gHOVXjwnzTz5uv 5E04rP/GwcWyVXJUdlB8mPIrNWaEgEue0MwSQdqMN4p17MR8e4l2aG7wyigpVuppPrM6 opEsZ/6WtZ6cKTX0mYy1VgUL14mJBzsidRPLReGWDmGUtJ1nyxJ+xJMhU71DnbA+UIr1 JZXeYlYOcXW6g58BwCMR56fzGKSYrTjDBCQoc+JS3v79KBS/S8QdcsTEKXMzLseX/7p4 XEIg== X-Gm-Message-State: AOJu0YyY3hylDvw/FgpuxA35jexTmqxCNAcR6acvL5p6xFlNdOH3n6j5 0Byl9ugJ32nUz/vWKBjLIPL7rLzIBdg= X-Google-Smtp-Source: AGHT+IE9CVDa0uPYwQ3dK4HiagUY1Wab+0CtaMHuuj9jitAwMCnxS1Hx84mPVFs+ql161c+jSgZwkA== X-Received: by 2002:a17:903:245:b0:1c5:d8a3:8789 with SMTP id j5-20020a170903024500b001c5d8a38789mr32291748plh.4.1699380618992; Tue, 07 Nov 2023 10:10:18 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:18 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 3/9] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Tue, 7 Nov 2023 10:07:07 -0800 Message-ID: <20231107180907.553451-4-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index f80dd6474f..707fd9fe7f 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -774,7 +774,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1149,7 +1149,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 6c1ccda159..754c885cd1 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -762,7 +762,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -785,7 +785,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -1008,7 +1008,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1188,7 +1188,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 334c51fddb..de6469eef7 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -232,7 +232,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; bool is_dcd; From patchwork Tue Nov 7 18:07:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861207 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=fvcr7i3K; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPxBF1wK8z1yQL for ; Wed, 8 Nov 2023 05:11:49 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXJ-0004gN-Ta; Tue, 07 Nov 2023 13:10:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QXI-0004e6-0P for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:48 -0500 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWs-0002jk-IT for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:47 -0500 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1cc0e78ec92so40281875ad.3 for ; Tue, 07 Nov 2023 10:10:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380620; x=1699985420; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cIc7H0VDI7Su5JGUvQgueNQKhCP3W7elx90e94K+TnQ=; b=fvcr7i3Kkh4dEJmCJH+Ly8RhBOAJ+a83YOxm+frHUR6tlb0F2FSKubnFni8beDXK6J IgHEj+BnbLNjoMrYwhYlDTNvJpaOR1SLUAVkQnnWJaZ+rv1G47j6dURSXc3G64Zetl7m TsBmx6K5Y1QXLfUZlDTOQNLxEIuI0VB37McythsP4O0H8pgKj6d0/mcxhHQaqOtJxFj/ 6vQWKTMlQdiEHWiqPKJYd5BNQu5fcXzi+RH+AGhYjo7YyB3pffuYcUCqS84USEIfevMt 6CWRKn+lSHfD59rpczEBAE4EXz4z4yqRFZSlpG1h0RgS3H/i9s+3o8qpzHXw58JKshJd d7Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380620; x=1699985420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cIc7H0VDI7Su5JGUvQgueNQKhCP3W7elx90e94K+TnQ=; b=aR/Zf7NNZS7eOB2x+W9Cr3rvtrFPeb0P32bzac7O7uHXCKIxSmAdi/cticArgKJqzv HPP0fzc+RxZY0GdB/ueqU2U4YJSHvhbYE2df3DXC4Ld1DBIyO5mU0yd3fxYY4TJGkOX9 3h/D4MQFFQqs9s9pD6rfOLQYBXg4UuryH65wxuhIf4mO6E9aWJUDNqbq7i+NEapyBUe7 7PKza/jMHNxoyM3TUVCZWBrt/FFt+Bq5erV91GDgkOY1hPJb8t5VwVhDEIEl/kRIe+Nh wApQYnLnXWCKUqM53CrHjHPtFFxwsn50Yduzmsw5/t0Jx2xaf6A/tbEymML3GwewVIT6 5vYQ== X-Gm-Message-State: AOJu0YyZn7v2ccpuWJvz4Gwo1Abg8epWd+24oWDMSaZVk1j1K6rRARlb DIYN0K+yAuw0+pvrzJ7fdaQMT86sSz4= X-Google-Smtp-Source: AGHT+IHGNH7ZWpcEDNR+g0tlYCx5gQyhkoR95XBku5YzVtfRDOD5hxW6GKGgVBtcG5+1BC6GyRPz2w== X-Received: by 2002:a17:902:a418:b0:1cc:2eda:bde6 with SMTP id p24-20020a170902a41800b001cc2edabde6mr26518551plq.34.1699380619857; Tue, 07 Nov 2023 10:10:19 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:19 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 4/9] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Tue, 7 Nov 2023 10:07:08 -0800 Message-ID: <20231107180907.553451-5-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::631; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x631.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support as below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 754c885cd1..2d67d2015c 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -721,6 +721,36 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +static int cxl_create_dc_regions(CXLType3Dev *ct3d) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 8; /* 8*256MB */ + uint64_t blk_size = 2 * MiB; + CXLDCDRegion *region; + + if (ct3d->hostvmem) { + region_base += ct3d->hostvmem->size; + } + if (ct3d->hostpmem) { + region_base += ct3d->hostpmem->size; + } + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + region->base = region_base; + region->decode_len = decode_len; + region->len = region_len; + region->block_size = blk_size; + /* dsmad_handle is set when creating cdat table entries */ + region->flags = 0; + + region_base += region->len; + } + + return 0; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -789,6 +819,10 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (cxl_create_dc_regions(ct3d)) { + return false; + } + return true; } @@ -1108,6 +1142,7 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT16("spdm", CXLType3Dev, spdm_port, 0), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Tue Nov 7 18:07:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861209 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=TsnQ8RjA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPxCT3wW4z1yQL for ; Wed, 8 Nov 2023 05:12:53 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXQ-0005LB-1m; Tue, 07 Nov 2023 13:10:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QXO-0005Cb-66 for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:54 -0500 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWx-0002k1-MU for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:53 -0500 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1cc3216b2a1so48538895ad.2 for ; Tue, 07 Nov 2023 10:10:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380621; x=1699985421; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fidJGI9tSfEUK7LuKCCNr/HhdNthEOLs0OgUT171rXg=; b=TsnQ8RjAzqCgqUp3l7dAXTXZXdj0CuZ0LbxEyU6SDTmNoMkHyvZwM2a2ieRBq4bpH+ JjJWFVGYcSd2o7lSxO9+dyV6ku892dmqBTrAtQcESHJ5tAMVEC/7K14l+oEmZ5MmTURQ 8x4Zj7gAR8gmCOfI/gSCFygQ50sOGoVNznrj68/tre8gGpOLP3ldzX2YGkgCZ0OpuvFC ZrzjgCmqGV29qh9iCbwRSmXS6IRkz/tRHUG+/9XRzAldIZxxMyMjQQtcwn5kBd1tsUvR nhxOEHnBi4ML6bwa8iN7myVaEAW0Ix+Ym/R8WMND2aGy4nDFVcyRZ2y7lcv7eboyu1Nh wp5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380621; x=1699985421; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fidJGI9tSfEUK7LuKCCNr/HhdNthEOLs0OgUT171rXg=; b=GxgHBwJq05vwXac34g5ORkcKwjSxl+wn6XpJeGseFfeSuXemf+yfvWQFYjPsm69t4v tIRM0sQaTaK9VUdnDgq22luqSSaDipxRvn/eP5QXo/MApaKdk72NyV5B/FOK8rY9egp5 YM9gFdFZpNPVepDMuQxDcUDZdpsvYG7vy9bZUZAZTUN4ifmQSkHJBUKfSSDO3i9cFFCy SGAkQ+ElrkvzhMLH+YaWAL/EilIPZqGOdFXxiuHRGbzJBxaR9v4fYIZ50hYuUeHTeeHX 1JCu5wAWwrpiED1IJg6fUqT1Zsv60/6l8D37bbAqXLTfJQk7x0N61MvsOnAdSvs/r5QS WkPw== X-Gm-Message-State: AOJu0YxtPqbCa/BxfaUvmIbzxowkiOESspmSeZG88da4dcS7g5YR/wf3 lVBA50sD8nzztVj5aMwURUTI7XxUfiM= X-Google-Smtp-Source: AGHT+IH2sOSQhzqP6upCTCtWBH3TOQDDXqGUL3eGXNC+4Wz3oTZeRcj6fBK6yv5oVwcRAzQmxyDd9g== X-Received: by 2002:a17:903:11c3:b0:1cc:5549:aabd with SMTP id q3-20020a17090311c300b001cc5549aabdmr27205343plh.8.1699380621077; Tue, 07 Nov 2023 10:10:21 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:20 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 5/9] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Tue, 7 Nov 2023 10:07:09 -0800 Message-ID: <20231107180907.553451-6-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Add (file/memory backed) host backend, all the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, following supports are added: 1. Add a new property to type3 device "nonvolatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all dc regions share one one host backend. 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region; 4. Fix dvsec range registers to include DC regions. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 16 ++- hw/mem/cxl_type3.c | 198 +++++++++++++++++++++++++++++------- include/hw/cxl/cxl_device.h | 4 + 3 files changed, 179 insertions(+), 39 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 707fd9fe7f..1f512b3e6b 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -596,7 +596,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -610,7 +611,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -764,7 +766,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -805,9 +808,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1149,7 +1154,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 2d67d2015c..152a51306d 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -31,6 +31,7 @@ #include "hw/pci/spdm.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -44,8 +45,9 @@ enum { }; static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, - bool is_pmem, uint64_t dpa_base) + int dsmad_handle, uint64_t size, + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; g_autofree CDATDslbis *dslbis0 = NULL; @@ -64,9 +66,10 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -150,7 +153,7 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -169,21 +172,28 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; int dsmad_handle = 0; int cur_ent = 0; int len = 0; int rc, i; + uint64_t vmr_size = 0, pmr_size = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } + if (ct3d->hostpmem && ct3d->hostvmem && ct3d->dc.host_dc) { + warn_report("The device has static ram and pmem and dynamic capacity"); + } + if (ct3d->hostvmem) { volatile_mr = host_memory_backend_get_memory(ct3d->hostvmem); if (!volatile_mr) { return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -192,6 +202,19 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); + } + + if (ct3d->dc.num_regions) { + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } else { + return -EINVAL; + } } table = g_malloc0(len * sizeof(*table)); @@ -201,8 +224,8 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) /* Now fill them in */ if (volatile_mr) { - rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, - false, 0); + rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, + false, false, 0); if (rc < 0) { return rc; } @@ -210,14 +233,38 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; rc = ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, false, vmr_size); if (rc < 0) { goto error_cleanup; } cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + uint64_t region_base = vmr_size + pmr_size; + + /* + * Currently we create cdat entries for each region, should we only + * create dsmas table instead?? + * We assume all dc regions are non-volatile for now. + * + */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + rc = ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + true, true, region_base); + if (rc < 0) { + goto error_cleanup; + } + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -445,11 +492,24 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_hi = ct3d->hostpmem->size >> 32; range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); + } + } else { + range1_size_hi = ct3d->dc.host_dc->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -721,6 +781,9 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: region parameters are hard coded, may need to change in the future. + */ static int cxl_create_dc_regions(CXLType3Dev *ct3d) { int i; @@ -736,6 +799,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) if (ct3d->hostpmem) { region_base += ct3d->hostpmem->size; } + for (i = 0; i < ct3d->dc.num_regions; i++) { region = &ct3d->dc.regions[i]; region->base = region_base; @@ -755,7 +819,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -823,6 +888,50 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) return false; } + ct3d->dc.total_capacity = 0; + if (ct3d->dc.host_dc) { + MemoryRegion *dc_mr; + char *dc_name; + uint64_t total_region_size = 0; + int i; + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have backing device"); + return false; + } + /* FIXME: set dc as nonvolatile for now */ + memory_region_set_nonvolatile(dc_mr, true); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + + for (i = 0; i < ct3d->dc.num_regions; i++) { + total_region_size += ct3d->dc.regions[i].len; + } + /* Make sure the host backend is large enough to cover all dc range */ + if (total_region_size > memory_region_size(dc_mr)) { + error_setg(errp, + "too small host backend size, increase to %lu MiB or more", + total_region_size / MiB); + return false; + } + + if (dc_mr->size % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to %lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + + ct3d->dc.total_capacity = total_region_size; + } + return true; } @@ -933,6 +1042,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -952,6 +1064,9 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); spdm_sock_fini(ct3d->doe_spdm.socket); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -1025,16 +1140,24 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + /* Do we want dc_size to be dc_mr->size or not?? */ + dc_size = ct3d->dc.total_capacity; } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -1042,19 +1165,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -1143,6 +1265,8 @@ static Property ct3_props[] = { DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT16("spdm", CXLType3Dev, spdm_port, 0), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("nonvolatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1209,33 +1333,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = ct3d->dc.total_capacity; + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index de6469eef7..3dc6928bc5 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,10 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCDRegion regions[DCD_MAX_REGION_NUM]; } dc; From patchwork Tue Nov 7 18:07:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861211 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=eKlNEv12; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPxDN57FHz1yQL for ; Wed, 8 Nov 2023 05:13:40 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXE-0004aU-64; Tue, 07 Nov 2023 13:10:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QXC-0004Xb-7W for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:42 -0500 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWv-0002kQ-Ju for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:41 -0500 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-6b2018a11efso6109427b3a.0 for ; Tue, 07 Nov 2023 10:10:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380622; x=1699985422; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nUVbPa2RXh5sULa717dlwLdK6nK0VOlZVO7MX7gSRww=; b=eKlNEv12fdPHqW6ll3fCfK7NwsPsfSa3Mx9o29w92D6MFUGqZ00xhRVDyzW7ho4IgM 80imsGxcCWRWx/UsHdb8MiHPUQE6W8MDFZLx1OT1wMP3oWSBT+MwiaB+1ttg/1EC+rB0 6jrEsKQfPr9t9hsL+md6nTnoFbpcdeCmkpx3y6XQryXAEno/GpPqUiPO90VRVl/I7LQW avc53sXtF84/0yHQqWt1q0TIN5cUJzVEsliBIRJihKr6hkeDlCC1oDCHYIxXc5QZ4O5U /qGP9eLquLLViTqNwdCKzkcUTa9HcJlnee+K9l8cYKR+bQH3KPtzw9LiRVeI9Q2D2eSi jHgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380622; x=1699985422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nUVbPa2RXh5sULa717dlwLdK6nK0VOlZVO7MX7gSRww=; b=r3CFyOeAx4hK8w1IAEQJg5Jamzc7mQKyq1cmcvn9mkeoR2y2yCFWoOJPk62XQx5lJS IxMhUGFBLem7KOo/zQezEXYo5Aad47qJQRCZFCwwTk6I7jySwQgCBi+v/thKXu4S7BPL OFxe3D2ufMSVkdURpIKigYex/AvO0scBIlVpvempTRj7kIg442MoCShgA+e2dcEelKYk qWx+cHuTvPIa9dJSMgdhHJwrcZWGTrg0EKxz5PVkxUcAMhCCCzS+3wu+uKqqkPm7QCD5 07vfUOsoS9QSC5EXdhp27fZwCX8m694BfxPVabKQxFtBdR2juw1Ra2PX5aSgfugnzW1E 3aJw== X-Gm-Message-State: AOJu0YxjHQJwWRloV/hQDHs9p+idW+3FSgLzws4EPG6BFJOnYpeG3WX1 qT4l20MfClgWLNEs+tdtH9ODCUxqRgA= X-Google-Smtp-Source: AGHT+IHbIav+SSkCelT+QQDr+/PZe04ppInU4J8uSOtgCzMCmlf2Dg5sNGz15p4xGCJORRfIXBtO/A== X-Received: by 2002:a05:6a20:3d82:b0:16b:9886:7eda with SMTP id s2-20020a056a203d8200b0016b98867edamr38574558pzi.35.1699380622013; Tue, 07 Nov 2023 10:10:22 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:21 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 6/9] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Tue, 7 Nov 2023 10:07:10 -0800 Message-ID: <20231107180907.553451-7-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::429; envelope-from=nifan.cxl@gmail.com; helo=mail-pf1-x429.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and add get DC extent list mailbox command per CXL.spec.3.0:.8.2.9.8.9.2. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 73 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 23 ++++++++++++ 3 files changed, 97 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 1f512b3e6b..56f4aa237a 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -82,6 +82,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1286,6 +1287,75 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.0 section 8.2.9.8.9.2: + * Get Dynamic Capacity Extent List (Opcode 4810h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct get_dyn_cap_ext_list_in_pl { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED; + + struct get_dyn_cap_ext_list_out_pl { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED; + + struct get_dyn_cap_ext_list_in_pl *in = (void *)payload_in; + struct get_dyn_cap_ext_list_out_pl *out = (void *)payload_out; + uint16_t record_count = 0, i = 0, record_done = 0; + CXLDCDExtentList *extent_list = &ct3d->dc.extents; + CXLDCDExtent *ent; + uint16_t out_pl_len; + uint32_t start_extent_id = in->start_extent_id; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + /* May need more processing here in the future */ + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + memset(out, 0, out_pl_len); + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out->records[record_done].start_dpa, ent->start_dpa); + stq_le_p(&out->records[record_done].len, ent->len); + memcpy(&out->records[record_done].tag, ent->tag, 0x10); + stw_le_p(&out->records[record_done].shared_seq, ent->shared_seq); + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1333,6 +1403,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 152a51306d..c9d792a725 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -811,6 +811,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) region_base += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return 0; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 3dc6928bc5..5738c6f434 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -420,6 +420,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_REGION_NUM 8 +typedef struct CXLDCDExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCDExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCDExtent) node; +} CXLDCDExtent; +typedef QTAILQ_HEAD(, CXLDCDExtent) CXLDCDExtentList; + typedef struct CXLDCDRegion { uint64_t base; uint64_t decode_len; /* aligned to 256*MiB */ @@ -470,6 +489,10 @@ struct CXLType3Dev { HostMemoryBackend *host_dc; AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ + CXLDCDExtentList extents; + + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCDRegion regions[DCD_MAX_REGION_NUM]; From patchwork Tue Nov 7 18:07:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861206 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=hnxzGP9w; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPx9v3yX0z1yQL for ; Wed, 8 Nov 2023 05:11:31 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXB-0004We-68; Tue, 07 Nov 2023 13:10:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QX9-0004Td-MJ for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:39 -0500 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWu-0002ke-Po for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:39 -0500 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1cc5b6d6228so40268455ad.2 for ; Tue, 07 Nov 2023 10:10:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380623; x=1699985423; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qUg/SG54upIHtQz6IlNDfZW8H+6HjCMvwHk2jyDUo0c=; b=hnxzGP9wcmKF4umBqfLSLST5qg/rmCbgkz42ONPPKQ7jGlwT4+uKOuZUQglLHscCpy PAUNZb2Xe39zpbKTmw5sX1tuE550rXESsElH4Se1Cnul5Nspm6kknJEIbU62TlyQ9PXg qeNhwqxF0QL9KxIBtGX3Z66C5uR3lRAUId1Bs7E1V3bw0JCMgGBH+UECbef8fDHRwuk4 uTXjGuj3i7A1PnGop/LoDG9Wo7PXD9DLEvKkTYqGR5ci5pVw/EjqJ6mwruhePvh5X9OB 1Ew5UnVNgZ3nIxN5dnjNZvc5Xw9MzaguH/oPuFWuwZTnStYQCHl/ScsobrKPdlTqqeSy N/Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380623; x=1699985423; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qUg/SG54upIHtQz6IlNDfZW8H+6HjCMvwHk2jyDUo0c=; b=MEC64Sem39NjbILTC+z9UvPY2DXdaNuyEt9MvhKd++urASR9BfRe04b86sBN6U3JWD Y8ZvI9lpwUsFQn5kEhZshjPfssOo7zpO+lt32Q8wUsAjOKHJyzNOJT98mVStUXF+NCfv M7L/d8C6g8vgy0lnZAk7r5DObsRtRknWHGzVngyYWtB8XqbpGqXtqK6d60uMLYRV8U8v R5A0Ph8vB0pOzVm7+tn7ozji/5z97UIvnD/aQmMdqk3GDH5oZUlS9pxk/qwRQJVhQJoe c4W7XmL4veR+0X6K9T+jQAbTcdUPvYBSNif8UnzuTsbMvWXc1lcJpCxcYXVARsY3J4Om k/5Q== X-Gm-Message-State: AOJu0YyVtY0mvJeKAsKkkCP5Ed6kzNk8pQ4JDkkUBLE0MrnZb6/wDeAs ufoJbIU64psdsQBLyj+f3l88f3MQPNI= X-Google-Smtp-Source: AGHT+IHquAmXxcjrVWrbCx9nCMe2dbsGzG8XJwXYiCLO2ckHcrK2QKzgao7HwzetjQuOJ44GGFU3pg== X-Received: by 2002:a17:903:1210:b0:1c9:e4f2:a39d with SMTP id l16-20020a170903121000b001c9e4f2a39dmr34722916plh.49.1699380622885; Tue, 07 Nov 2023 10:10:22 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:22 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 7/9] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Tue, 7 Nov 2023 10:07:11 -0800 Message-ID: <20231107180907.553451-8-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::630; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x630.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Per CXL spec 3.0, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.8.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.8.9.4. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 271 ++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 3 +- include/hw/cxl/cxl_device.h | 5 +- 3 files changed, 277 insertions(+), 2 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 56f4aa237a..9f788b03b6 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -83,6 +83,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1356,6 +1358,269 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, int nr, int size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + CXLDCDRegion *region = &ct3d->dc.regions[0]; + int i; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.0 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + /*Should we compare with decode_len or len of the region??*/ + if (dpa + len > region->base + + region->decode_len * CXL_CAPACITY_MULTIPLIER) + return NULL; + return region; + } + } + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCDExtent *extent; + + extent = g_new0(CXLDCDExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } else { + memset(extent->tag, 0, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +/* + * CXL r3.0 Table 8-129: Add Dynamic Capacity Response Input Payload + * CXL r3.0 Table 8-131: Release Dynamic Capacity Input Payload + */ +typedef struct updated_dc_extent_list_in_pl { + uint32_t num_entries_updated; + uint8_t rsvd[4]; + /* CXL r3.0 Table 8-130: Updated Extent List */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED updated_dc_extent_list_in_pl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const updated_dc_extent_list_in_pl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCDRegion *region = &ct3d->dc.regions[0]; + CXLDCDRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->len + lastregion->base - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.0 section 8.2.9.8.9.3: Add Dynamic Capacity Response (opcode 4802h) + * + * Assume an extent is added only after the response is processed successfully + * TODO: for better extent list validation, a better solution would be + * maintaining a pending extent list and use it to verify the extent list in + * the response. + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + updated_dc_extent_list_in_pl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCDExtentList *extent_list = &ct3d->dc.extents; + CXLDCDExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_SUCCESS; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + /* + * Check if the DPA range of the to-be-added extent overlaps with + * existing extent list maintained by the device. + */ + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + return CXL_MBOX_INVALID_PA; + /* Overlapping one end of the other */ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_PA; + } + } + + /* + * TODO: add a pending extent list based on event log record and + * verify the input response + */ + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.0 section 8.2.9.8.9.4: Release Dynamic Capacity (opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + updated_dc_extent_list_in_pl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCDExtentList *extent_list = &ct3d->dc.extents; + CXLDCDExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + /* Remove any partial extents */ + uint64_t len1 = dpa - ent->start_dpa; + uint64_t len2 = ent->start_dpa + ent->len - dpa - len; + + if (len1) { + cxl_insert_extent_to_extent_list(extent_list, + ent->start_dpa, len1, + NULL, 0); + ct3d->dc.total_extent_count += 1; + } + if (len2) { + cxl_insert_extent_to_extent_list(extent_list, dpa + len, + len2, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + break; + /*Currently we reject the attempt to remove a superset*/ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + } + + if (ent) { + QTAILQ_REMOVE(extent_list, ent, node); + g_free(ent); + ct3d->dc.total_extent_count -= 1; + } else { + /* Try to remove a non-existing extent */ + return CXL_MBOX_INVALID_PA; + } + } + + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1406,6 +1671,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "ADD_DCD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "RELEASE_DCD_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c9d792a725..482329a499 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -789,7 +789,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) int i; uint64_t region_base = 0; uint64_t region_len = 2 * GiB; - uint64_t decode_len = 8; /* 8*256MB */ + uint64_t decode_len = 2 * GiB; uint64_t blk_size = 2 * MiB; CXLDCDRegion *region; @@ -803,6 +803,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) for (i = 0; i < ct3d->dc.num_regions; i++) { region = &ct3d->dc.regions[i]; region->base = region_base; + /* NOTE: Should be divided by 256 * MiB before be returned to host */ region->decode_len = decode_len; region->len = region_len; region->block_size = blk_size; diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 5738c6f434..b3d35fe000 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -130,7 +130,8 @@ typedef enum { CXL_MBOX_INCORRECT_PASSPHRASE = 0x14, CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15, CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16, - CXL_MBOX_MAX = 0x17 + CXL_MBOX_INVALID_EXTENT_LIST = 0x1E, /* cxl r3.0: Table 8-34*/ + CXL_MBOX_MAX = 0x1F } CXLRetCode; typedef struct CXLCCI CXLCCI; @@ -548,4 +549,6 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + #endif From patchwork Tue Nov 7 18:07:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861210 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=c9F+m7fV; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPxD320RSz1yQL for ; Wed, 8 Nov 2023 05:13:23 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXG-0004dj-SX; Tue, 07 Nov 2023 13:10:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QXC-0004Xv-Ir for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:42 -0500 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWv-0002kk-Ky for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:42 -0500 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1cc3216b2a1so48539255ad.2 for ; Tue, 07 Nov 2023 10:10:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380624; x=1699985424; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4lhR5l4PvKeDTkn4MSXEnhevDWdoFM18L2lwlXsuKYY=; b=c9F+m7fVJ9+81U+Y59UJcEYRBvcbE4G01gZ9JLs0ImDgRNQehR96WIICbxxKAZVF6v rq/9bgx7YAe9Yco2GvBH+G6fihTNmKuwqIXwxt+vSrRxzLYYQqQYRmcMet/fOisiKW5Q MUzZAGoF43lK4Fr3Av8RFT+10XLhEDcQSidfwwvqtz3f3VdlVmIGxK5SHKDFsfVBqGCE uHmkwRoS9tV59nqAvm6JZxLwK23t/mTFE0gg3jFFxr1nzcQzBrwM+JhDfaqycmWffGxl LvIep9fNiKudaFSPmF4ec18twVs4AazNY2xVXQnFCW+M44u64IYlUtRi1T69DTjzoDnw c12Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380624; x=1699985424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4lhR5l4PvKeDTkn4MSXEnhevDWdoFM18L2lwlXsuKYY=; b=JUOZrby1pfQWUa+eYKBLRUTJ1dCZZPXdnWREXJS0wYNOp+cFY+oXiFTpUZGVPr+Ygh lGFpWhEnwUYr6IdjqpRaF+crO4e87cdmmNOyUqIcTlMxh6IR3M348fm+XMXTqIrTc4UT Ai+pjQ8Px/ve2b8Ox7cO9TRU1nXSKktRcORcapQZdU/X3K+bpiqlxs1tLwWLCO+gFojX n6GVtMuchQj5ReA/mKgAFlK+r7Ci5Ca5x6erH7kl8bCTiu9YPp4Uc2OpJrpysniX7ii0 F/4MtpC7IEvLo4X/SLyUhhRg70EYTBDdriDjHb4KyqRLCCbEp08sUbRIpNd1XVTGShUf R6Qw== X-Gm-Message-State: AOJu0YzFPO1F/aaCsbiDqjTM814mkO4D6gFEA2Cb46Yk6ScC10/qNNWW O0gmisvu9G//BA2iw/TEM8NqxTy71vE= X-Google-Smtp-Source: AGHT+IH7Yev1hBFrxPbTSy5qpYNKvsSQwQ3Ee8RWNDp/nsCivGGFhd0KNLBE/oO1T2WzjmzRvT5sJQ== X-Received: by 2002:a17:902:e84b:b0:1cc:3825:7ba2 with SMTP id t11-20020a170902e84b00b001cc38257ba2mr29811079plg.4.1699380623831; Tue, 07 Nov 2023 10:10:23 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:23 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 8/9] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Tue, 7 Nov 2023 10:07:12 -0800 Message-ID: <20231107180907.553451-9-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::632; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x632.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Since fabric manager emulation is not supported yet, the change implements the functions to add/release dynamic capacity extents as QMP interfaces. Note: we block any FM issued extent release request if the exact extent does not exist in the extent list of the device. We will loose the restriction later once we have partial release support in the kernel. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 0, "len": 128 }, { "dpa": 128, "len": 128 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) look like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 128, "len": 128 } ] } } Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 25 +++- hw/mem/cxl_type3.c | 225 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 14 +++ include/hw/cxl/cxl_device.h | 8 +- include/hw/cxl/cxl_events.h | 15 +++ qapi/cxl.json | 60 +++++++++- 6 files changed, 338 insertions(+), 9 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 9f788b03b6..8e6a98753a 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1362,7 +1362,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, int nr, int size) +bool test_any_bits_set(const unsigned long *addr, int nr, int size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1400,7 +1400,7 @@ CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1538,15 +1538,28 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, } } - /* - * TODO: add a pending extent list based on event log record and - * verify the input response - */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents_pending_to_add, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + break; + } + } + if (ent) { + QTAILQ_REMOVE(&ct3d->dc.extents_pending_to_add, ent, node); + g_free(ent); + } else { + return CXL_MBOX_INVALID_PA; + } cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; } + /* + * TODO: extents_pending_to_add needs to be cleared so the extents not + * accepted can be reclaimed base on spec r3.0: 8.2.9.8.9.3 + */ + return CXL_MBOX_SUCCESS; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 482329a499..43cea3d818 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -813,6 +813,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) region_base += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending_to_add); return 0; } @@ -1616,7 +1617,8 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ + case CXL_EVENT_LOG_DYNCAP: + return CXL_EVENT_TYPE_DYNAMIC_CAP; default: return -EINVAL; } @@ -1867,6 +1869,227 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.0 Table 8-47: Dynanic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, + DC_EVENT_NUM +} CXLDCEventType; + +/* + * Check whether the exact extent exists in the list + * Return value: true if exists, otherwise false + */ +static bool cxl_dc_extent_exists(CXLDCDExtentList *list, CXLDCExtentRaw *ext) +{ + CXLDCDExtent *ent; + + if (!ext || !list) { + return false; + } + + QTAILQ_FOREACH(ent, list, node) { + if (ent->start_dpa != ext->start_dpa) { + continue; + } + + /*Found exact extent*/ + if (ent->len == ext->len) { + return true; + } else { + return false; + } + } + return false; +} + +static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, + CXLDCEventType type, uint16_t hid, + uint8_t rid, + CXLDCExtentRecordList *records, + Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDCExtentRecordList *list; + g_autofree CXLDCExtentRaw *extents = NULL; + CXLDCDExtentList *extent_list = NULL; + uint8_t enc_log; + uint64_t offset, len, block_size; + int i; + int rc; + g_autofree unsigned long *blk_bitmap = NULL; + + obj = object_resolve_path(path, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve path"); + return; + } + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + error_setg(errp, "Path not point to a valid CXL type3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + rc = ct3d_qmp_cxl_event_log_enc(log); + if (rc < 0) { + error_setg(errp, "Unhandled error log type"); + return; + } + enc_log = rc; + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset * MiB; + len = list->value->len * MiB; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + num_extents++; + list = list->next; + } + if (num_extents == 0) { + error_setg(errp, "No extents found in the command"); + return; + } + + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Create Extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + offset = list->value->offset * MiB; + len = list->value->len * MiB; + + extents[i].start_dpa = offset + dcd->dc.regions[rid].base; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + + /* + * We block the release request from FM if the exact extent has + * not been accepted by the host yet + * TODO: We can loose the restriction by skipping the check if desired + */ + if (type == DC_EVENT_RELEASE_CAPACITY || + type == DC_EVENT_FORCED_RELEASE_CAPACITY) { + if (!cxl_dc_extent_exists(&dcd->dc.extents, &extents[i])) { + error_setg(errp, "No exact extent found in the extent list"); + return; + } + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + list = list->next; + i++; + } + + switch (type) { + case DC_EVENT_ADD_CAPACITY: + extent_list = &dcd->dc.extents_pending_to_add; + break; + default: + break; + } + /* + * CXL r3.0 section 8.2.9.1.5: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (extent_list) { + cxl_insert_extent_to_extent_list(extent_list, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_ADD_CAPACITY, 0, + region_id, records, errp); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_RELEASE_CAPACITY, 0, + region_id, records, errp); +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..d913b11b4d 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,17 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index b3d35fe000..ca4f824b11 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -491,6 +491,7 @@ struct CXLType3Dev { AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ CXLDCDExtentList extents; + CXLDCDExtentList extents_pending_to_add; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -550,5 +551,10 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); - +void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, int nr, int size); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index d778487b7e..4f8cb3215d 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,19 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.0 section Table 8-47: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t reserved1; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t reserved2[3]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x20]; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 8cc4c72fa9..6b631f64f1 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -25,7 +25,8 @@ 'data': ['informational', 'warning', 'failure', - 'fatal'] + 'fatal', + 'dyncap'] } ## @@ -361,3 +362,60 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDCExtentRecord: +# +# Record of a single extent to add/release +# +# @offset: offset of the extent start related to current region base address +# @len: extent size (in MiB) +# +# Since: 8.0 +## +{ 'struct': 'CXLDCExtentRecord', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to start add dynamic capacity extents flow. The host will +# need to respond to indicate it accepts the capacity before it becomes +# available for read and write. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add/release +# @extents: Extents to add +# +# Since : 8.2 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to start release dynamic capacity extents flow. The host will +# need to respond to indicate that it has released the capacity before it +# is made unavailable for read and write and can be re-added. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add/release +# @extents: Extents to release +# +# Since : 8.2 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} From patchwork Tue Nov 7 18:07:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: fan X-Patchwork-Id: 1861212 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=MSmhtCTT; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SPxDZ4mjSz1yQL for ; Wed, 8 Nov 2023 05:13:50 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r0QXG-0004dG-6t; Tue, 07 Nov 2023 13:10:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r0QXE-0004Zj-1k for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:44 -0500 Received: from mail-pj1-x102f.google.com ([2607:f8b0:4864:20::102f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r0QWx-0002l2-AZ for qemu-devel@nongnu.org; Tue, 07 Nov 2023 13:10:43 -0500 Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-2801d7f46f9so5004095a91.3 for ; Tue, 07 Nov 2023 10:10:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380625; x=1699985425; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z3VD1vraDKH7t/1eP7lxzrPhMiE3bMoC5UEpP6LTwk8=; b=MSmhtCTTJyegQDXpyly+w6ElBg3bLXueg7ErO734s45maLwSuwPtFYGernOYPRDMVX +UuG9R25qKDwJDQmaqoSUKLaeNGr6c8rEyN+tjQMY9codTt4Y5Sjsyvu4G6djbx5Ob8a R4aHxl3b49KqiPMxguVeTk2trEgD5xnQZJ+iHwZcQTn9eFJqlggal0OCgs0vlYQ3tOLH inZHdxntzhaA9Cfd2mNvl6LwBKkx8dsbH90gPoJNcjWxOguKQ6aaS+4naHBbhO8f23P3 GZtFQPi23jhOUkQ/9vUDQMzRIrpb+o/HHlZjl/t4qAgo3L4tEQjYDDwp7+W6j6FWYxJM zO1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380625; x=1699985425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z3VD1vraDKH7t/1eP7lxzrPhMiE3bMoC5UEpP6LTwk8=; b=L510b7fhl1zlDVurZaMkM5F1a27z3qty5hiza7MIB0/leNhg1W/cBbPRur9dTQpof7 9jAEYRP9zGeykG5zDDTOIvZM/fshukDGF5+/JPalZvou7wqAqEJrN+9Uvzg2zJECHICl tjWC4dJ3SQ1vWTd3AyimtYoS1MtNfVDJzlhAIkkvz3F761kwUEIuA+nzhIbVFSRpUuT4 IbYvZDDqYUvkCfS1G74n+S4EpI7ZTY31IDpxUlaoub8ru3Ta2Vimaw9hVqGoWcVi6x++ H26PM2PDheXr3EPAeVauTjliYlAS5Tx/75ENidOOvODAmhygoQJ3v1ukSGozYRMOIY8W 03GQ== X-Gm-Message-State: AOJu0Ywlw+wpN8f6lF4ttgvcQkQxyHN5FlPPrZT7Hk1Nad2Bp/i4RtVO 0SNaiBwV5irz8Ch2M0n/Wozh41yTGgE= X-Google-Smtp-Source: AGHT+IEGtULtzP3zUAeEcHZAMhAMLm8y71Y+vs76AD4I/K8Ta3QaxjqhuJ8KDNsahiaEcfKxi25Q/A== X-Received: by 2002:a17:90a:6f84:b0:280:9074:eb3d with SMTP id e4-20020a17090a6f8400b002809074eb3dmr17621113pjk.22.1699380624853; Tue, 07 Nov 2023 10:10:24 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:24 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 9/9] hw/mem/cxl_type3: Add dpa range validation for accesses to dc regions Date: Tue, 7 Nov 2023 10:07:13 -0800 Message-ID: <20231107180907.553451-10-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::102f; envelope-from=nifan.cxl@gmail.com; helo=mail-pj1-x102f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fan Ni Not all dpa range in the dc regions is valid to access until an extent covering the range has been added. Add a bitmap for each region to record whether a dc block in the region has been backed by dc extent. For the bitmap, a bit in the bitmap represents a dc block. When a dc extent is added, all the bits of the blocks in the extent will be set, which will be cleared when the extent is released. Signed-off-by: Fan Ni --- JC changes: - Rebase on what will be next gitlab.com/jic23/qemu CXL staging tree. - Drop unnecessary handling of failed bitmap allocations. In common with most QEMU allocations they fail hard anyway. - Use previously factored out cxl_find_region() helper - Minor editorial stuff in comments such as spec version references according to the standard form I'm trying to push through the code. Picked up Jørgen's fix: https://lore.kernel.org/qemu-devel/d0d7ca1d-81bc-19b3-4904-d60046ded844@wdc.com/T/#u --- hw/cxl/cxl-mailbox-utils.c | 31 +++++++++------ hw/mem/cxl_type3.c | 78 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 15 +++++-- 3 files changed, 109 insertions(+), 15 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 8e6a98753a..6be92fb5ba 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1401,10 +1401,9 @@ CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) } void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, - uint64_t dpa, - uint64_t len, - uint8_t *tag, - uint16_t shared_seq) + uint64_t dpa, uint64_t len, + uint8_t *tag, + uint16_t shared_seq) { CXLDCDExtent *extent; @@ -1421,6 +1420,13 @@ void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, QTAILQ_INSERT_TAIL(list, extent, node); } +static void cxl_remove_extent_to_extent_list(CXLDCDExtentList *list, + CXLDCDExtent *ent) +{ + QTAILQ_REMOVE(list, ent, node); + g_free(ent); +} + /* * CXL r3.0 Table 8-129: Add Dynamic Capacity Response Input Payload * CXL r3.0 Table 8-131: Release Dynamic Capacity Input Payload @@ -1545,14 +1551,15 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, } } if (ent) { - QTAILQ_REMOVE(&ct3d->dc.extents_pending_to_add, ent, node); - g_free(ent); + cxl_remove_extent_to_extent_list(&ct3d->dc.extents_pending_to_add, + ent); } else { return CXL_MBOX_INVALID_PA; } cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } /* @@ -1601,16 +1608,22 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, uint64_t len1 = dpa - ent->start_dpa; uint64_t len2 = ent->start_dpa + ent->len - dpa - len; + cxl_remove_extent_to_extent_list(extent_list, ent); + ct3d->dc.total_extent_count -= 1; + ct3_clear_region_block_backed(ct3d, dpa, len); + if (len1) { cxl_insert_extent_to_extent_list(extent_list, ent->start_dpa, len1, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } if (len2) { cxl_insert_extent_to_extent_list(extent_list, dpa + len, len2, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } break; /*Currently we reject the attempt to remove a superset*/ @@ -1621,11 +1634,7 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, } } - if (ent) { - QTAILQ_REMOVE(extent_list, ent, node); - g_free(ent); - ct3d->dc.total_extent_count -= 1; - } else { + if (!ent) { /* Try to remove a non-existing extent */ return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 43cea3d818..4ec65a751a 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -810,6 +810,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) /* dsmad_handle is set when creating cdat table entries */ region->flags = 0; + region->blk_bitmap = bitmap_new(region->len / region->block_size); region_base += region->len; } QTAILQ_INIT(&ct3d->dc.extents); @@ -818,6 +819,17 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) return 0; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + int i; + struct CXLDCDRegion *region; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -1046,6 +1058,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -1068,6 +1081,7 @@ static void ct3_exit(PCIDevice *pci_dev) spdm_sock_fini(ct3d->doe_spdm.socket); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -1078,6 +1092,66 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len) to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether a DPA range [dpa, dpa + len) has been backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len) to be unbacked and inaccessible. This + * happens when a dc extent is return by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -1178,6 +1252,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index ca4f824b11..b71b09700a 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -447,6 +447,7 @@ typedef struct CXLDCDRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCDRegion; struct CXLType3Dev { @@ -552,9 +553,15 @@ void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, - uint64_t dpa, - uint64_t len, - uint8_t *tag, - uint16_t shared_seq); + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); bool test_any_bits_set(const unsigned long *addr, int nr, int size); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif