From patchwork Wed Jul 8 01:34:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1324842 X-Patchwork-Delegate: bmeng.cn@gmail.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=G01YrImZ; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B1hrg1yMwz9sQt for ; Wed, 8 Jul 2020 11:42:43 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 28364821E1; Wed, 8 Jul 2020 03:38:32 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="G01YrImZ"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id CA4E382013; Wed, 8 Jul 2020 03:38:00 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x134.google.com (mail-il1-x134.google.com [IPv6:2607:f8b0:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id EC17C81F93 for ; Wed, 8 Jul 2020 03:37:51 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x134.google.com with SMTP id e18so26998156ilr.7 for ; Tue, 07 Jul 2020 18:37:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nBzu40fYCfexlWVBpsdIwClFLfgFQzArUZ6TFLhwQnU=; b=G01YrImZOTFP8STsIyQvTF8xMIcxkR91csor6FLwQokrDnnUKLiLGCTKc/7yBCHe/z F0Nx78pM9ObbkzbsMWlpSWiGBsWbz71gwtCJEy0OrWDf0a0o+1MYoHF7yEOQjvhImutN QL9OLSEWOmH6twBZ2naQS4MS/FePnhqzk2+sA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nBzu40fYCfexlWVBpsdIwClFLfgFQzArUZ6TFLhwQnU=; b=RRM+3ohR0I6/zPGEiLa40zQ7Oc1pel0Z21D1vbDsONHYYVYs60bamNO9MONdyBeUCw MiT6ySo+pjmkr5oGYee3j7rmtBvEae/Jfd6sR/4utosi6NBkWTPbE0jQ+/IRT0fmt7Rk PZ2POv7uRDzxvN21K/oFDUfrWVyw49zbvskDXFxApFLss0BkfNsgD0W2WWIklSk3oNOg Y/40xNivHj38x/pZrcwB+qWbk+7AH3laOPy/JUFn83YBWXYJ8wXlEvYLeGSYKYocjGRa TaMbavmAZ8uVJrnAshI5GO0SaFj01v232IiMDtsJvBpb2AYzidrTFRE08Titxch9DlNa nCGQ== X-Gm-Message-State: AOAM531Rt2S+NCwZU7AVbv7JhN4TZ9kfMtEyJLRPy9SaQwWOYKkYgI/C QDsoBLRKL9IAE7uIwIFkcGyg6RoNbpoHFw== X-Google-Smtp-Source: ABdhPJztup3j3BN+AmW1UmiVejbqJ2vmuXZBb5JR2tIY/Vz+HtLPSKqUsu40dYkW5xvp4rrS4TXuwQ== X-Received: by 2002:a92:3311:: with SMTP id a17mr10333245ilf.210.1594172270432; Tue, 07 Jul 2020 18:37:50 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id d6sm12538047ioo.9.2020.07.07.18.37.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jul 2020 18:37:50 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Bin Meng , Wolfgang Wallner , Andy Shevchenko , Simon Glass Subject: [PATCH v4 24/25] x86: mp: Add more comments to the module Date: Tue, 7 Jul 2020 19:34:44 -0600 Message-Id: <20200708013446.2600256-21-sjg@chromium.org> X-Mailer: git-send-email 2.27.0.383.g050319c2ae-goog In-Reply-To: <20200708013446.2600256-1-sjg@chromium.org> References: <20200708013446.2600256-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Add a description of how this module works and also some missing function comments. Reviewed-by: Wolfgang Wallner Signed-off-by: Simon Glass Reviewed-by: Bin Meng --- (no changes since v3) Changes in v3: - Remove stray asterisk from comments - Drop mention of cpu_map which was handled in a previous patch Changes in v2: - Add a new patch with more comments arch/x86/cpu/mp_init.c | 91 ++++++++++++++++++++++++++++++++++++++- arch/x86/include/asm/mp.h | 14 +++++- 2 files changed, 102 insertions(+), 3 deletions(-) diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c index 56e07ecd8f..b7c8a22fc6 100644 --- a/arch/x86/cpu/mp_init.c +++ b/arch/x86/cpu/mp_init.c @@ -32,13 +32,99 @@ DECLARE_GLOBAL_DATA_PTR; +/* + * Setting up multiprocessing + * + * See https://www.intel.com/content/www/us/en/intelligent-systems/intel-boot-loader-development-kit/minimal-intel-architecture-boot-loader-paper.html + * + * Note that this file refers to the boot CPU (the one U-Boot is running on) as + * the BSP (BootStrap Processor) and the others as APs (Application Processors). + * + * This module works by loading some setup code into RAM at AP_DEFAULT_BASE and + * telling each AP to execute it. The code that each AP runs is in + * sipi_vector.S (see ap_start16) which includes a struct sipi_params at the + * end of it. Those parameters are set up by the C code. + + * Setting up is handled by load_sipi_vector(). It inits the common block of + * parameters (sipi_params) which tell the APs what to do. This block includes + * microcode and the MTTRs (Memory-Type-Range Registers) from the main CPU. + * There is also an ap_count which each AP increments as it starts up, so the + * BSP can tell how many checked in. + * + * The APs are started with a SIPI (Startup Inter-Processor Interrupt) which + * tells an AP to start executing at a particular address, in this case + * AP_DEFAULT_BASE which contains the code copied from ap_start16. This protocol + * is handled by start_aps(). + * + * After being started, each AP runs the code in ap_start16, switches to 32-bit + * mode, runs the code at ap_start, then jumps to c_handler which is ap_init(). + * This runs a very simple 'flight plan' described in mp_steps(). This sets up + * the CPU and waits for further instructions by looking at its entry in + * ap_callbacks[]. Note that the flight plan is only actually run for each CPU + * in bsp_do_flight_plan(): once the BSP completes each flight record, it sets + * mp_flight_record->barrier to 1 to allow the APs to executed the record one + * by one. + * + * CPUS are numbered sequentially from 0 using the device tree: + * + * cpus { + * u-boot,dm-pre-reloc; + * #address-cells = <1>; + * #size-cells = <0>; + * + * cpu@0 { + * u-boot,dm-pre-reloc; + * device_type = "cpu"; + * compatible = "intel,apl-cpu"; + * reg = <0>; + * intel,apic-id = <0>; + * }; + * + * cpu@1 { + * device_type = "cpu"; + * compatible = "intel,apl-cpu"; + * reg = <1>; + * intel,apic-id = <2>; + * }; + * + * Here the 'reg' property is the CPU number and then is placed in dev->req_seq + * so that we can index into ap_callbacks[] using that. The APIC ID is different + * and may not be sequential (it typically is if hyperthreading is supported). + * + * Once APs are inited they wait in ap_wait_for_instruction() for instructions. + * Instructions come in the form of a function to run. This logic is in + * mp_run_on_cpus() which supports running on any one AP, all APs, just the BSP + * or all CPUs. The BSP logic is handled directly in mp_run_on_cpus(), by + * calling the function. For the APs, callback information is stored in a + * single, common struct mp_callback and a pointer to this is written to each + * AP's slot in ap_callbacks[] by run_ap_work(). All APs get the message even + * if it is only for one of them. When an AP notices a message it checks whether + * it should call the function (see check in ap_wait_for_instruction()) and then + * does so if needed. After that it sets its slot to NULL to indicate it is + * done. + * + * While U-Boot is running it can use mp_run_on_cpus() to run code on the APs. + * An example of this is the 'mtrr' command which allows reading and changing + * the MTRRs on all CPUs. + * + * Before U-Boot exits it calls mp_park_aps() which tells all CPUs to halt by + * executing a 'hlt' instruction. That allows them to be used by Linux when it + * starts up. + */ + /* This also needs to match the sipi.S assembly code for saved MSR encoding */ -struct saved_msr { +struct __packed saved_msr { uint32_t index; uint32_t lo; uint32_t hi; -} __packed; +}; +/** + * struct mp_flight_plan - Holds the flight plan + * + * @num_records: Number of flight records + * @records: Pointer to each record + */ struct mp_flight_plan { int num_records; struct mp_flight_record *records; @@ -59,6 +145,7 @@ struct mp_callback { int logical_cpu_number; }; +/* Stores the flight plan so that APs can find it */ static struct mp_flight_plan mp_info; /* diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h index 5b8811832a..192aae55e6 100644 --- a/arch/x86/include/asm/mp.h +++ b/arch/x86/include/asm/mp.h @@ -36,6 +36,14 @@ typedef int (*mp_callback_t)(struct udevice *cpu, void *arg); * * Note that ap_call() and bsp_call() can be NULL. In the NULL case the * callback will just not be called. + * + * @barrier: Ensures that the BSP and AP don't run the flight record at the same + * time + * @cpus_entered: Counts the number of APs that have run this record + * @ap_call: Function for the APs to call + * @ap_arg: Argument to pass to @ap_call + * @bsp_call: Function for the BSP to call + * @bsp_arg: Argument to pass to @bsp_call */ struct mp_flight_record { atomic_t barrier; @@ -83,7 +91,11 @@ struct mp_flight_record { */ int mp_init(void); -/* Set up additional CPUs */ +/** + * x86_mp_init() - Set up additional CPUs + * + * @returns < 0 on error, 0 on success. + */ int x86_mp_init(void); /**