From patchwork Tue Jan 21 13:54:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liu, Jing2" X-Patchwork-Id: 1226256 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.intel.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 481z1Q4nZMz9sRk for ; Tue, 21 Jan 2020 17:20:42 +1100 (AEDT) Received: from localhost ([::1]:48922 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmu0-0007KY-5w for incoming@patchwork.ozlabs.org; Tue, 21 Jan 2020 01:20:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41950) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmtC-0007G3-LT for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1itmtB-0006Sh-KN for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:50 -0500 Received: from mga03.intel.com ([134.134.136.65]:35723) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1itmtB-0006Rf-Bg for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:49 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jan 2020 22:19:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,344,1574150400"; d="scan'208";a="278301891" Received: from hyperv-sh3.bj.intel.com ([10.240.193.95]) by fmsmga001.fm.intel.com with ESMTP; 20 Jan 2020 22:19:45 -0800 From: Jing Liu To: virtio-dev@lists.oasis-open.org Subject: [virtio-dev] [PATCH v2 1/5] virtio-mmio: Add feature bit for MMIO notification Date: Tue, 21 Jan 2020 21:54:29 +0800 Message-Id: <1579614873-21907-2-git-send-email-jing2.liu@linux.intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> References: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 134.134.136.65 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zha Bin , kvm@vger.kernel.org, Jing Liu , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Chao Peng , Liu Jiang Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" All the queues notifications use the same register on MMIO transport layer. Add a feature bit (39) for enhancing the notification capability. The detailed mechanism would be in next patch. Co-developed-by: Chao Peng Signed-off-by: Chao Peng Co-developed-by: Liu Jiang Signed-off-by: Liu Jiang Co-developed-by: Zha Bin Signed-off-by: Zha Bin Signed-off-by: Jing Liu --- content.tex | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/content.tex b/content.tex index d68cfaf..826bc7d 100644 --- a/content.tex +++ b/content.tex @@ -5810,6 +5810,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} in its device notifications. See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}. \end{description} + \item[VIRTIO_F_MMIO_NOTIFICATION(39)] This feature indicates + that the device supports enhanced notification mechanism on + MMIO transport layer. \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} @@ -5843,6 +5846,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} or partially reset, and even without re-negotiating VIRTIO_F_SR_IOV after the reset. +A driver SHOULD accept VIRTIO_F_MMIO_NOTIFICATION if it is offered. + \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} A device MUST offer VIRTIO_F_VERSION_1. A device MAY fail to operate further @@ -5872,6 +5877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} and presents a PCI SR-IOV capability structure, otherwise it MUST NOT offer VIRTIO_F_SR_IOV. +If VIRTIO_F_MMIO_NOTIFICATION has been negotiated, a device +MUST support handling the notification from driver at the +calculated location. + \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits} Transitional devices MAY offer the following: From patchwork Tue Jan 21 13:54:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liu, Jing2" X-Patchwork-Id: 1226257 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.intel.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 481z3K3LBPz9sP3 for ; Tue, 21 Jan 2020 17:22:21 +1100 (AEDT) Received: from localhost ([::1]:48950 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmvb-0001NJ-83 for incoming@patchwork.ozlabs.org; Tue, 21 Jan 2020 01:22:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41963) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmtD-0007HG-QK for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1itmtC-0006T1-3R for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:51 -0500 Received: from mga03.intel.com ([134.134.136.65]:35723) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1itmtB-0006Rf-Ro for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:50 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jan 2020 22:19:48 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,344,1574150400"; d="scan'208";a="278301959" Received: from hyperv-sh3.bj.intel.com ([10.240.193.95]) by fmsmga001.fm.intel.com with ESMTP; 20 Jan 2020 22:19:47 -0800 From: Jing Liu To: virtio-dev@lists.oasis-open.org Subject: [virtio-dev] [PATCH v2 2/5] virtio-mmio: Enhance queue notification support Date: Tue, 21 Jan 2020 21:54:30 +0800 Message-Id: <1579614873-21907-3-git-send-email-jing2.liu@linux.intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> References: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 134.134.136.65 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zha Bin , kvm@vger.kernel.org, Jing Liu , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Chao Peng , Liu Jiang Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" With VIRTIO_F_MMIO_NOTIFICATION feature bit offered, the notification mechanism is enhanced. Driver reads QueueNotify register to get notification structure and calculate notification addresses of each virtqueue. Co-developed-by: Chao Peng Signed-off-by: Chao Peng Co-developed-by: Liu Jiang Signed-off-by: Liu Jiang Co-developed-by: Zha Bin Signed-off-by: Zha Bin Signed-off-by: Jing Liu --- content.tex | 53 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 13 deletions(-) diff --git a/content.tex b/content.tex index 826bc7d..5881253 100644 --- a/content.tex +++ b/content.tex @@ -1671,20 +1671,18 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi accesses apply to the queue selected by writing to \field{QueueSel}. } \hline - \mmioreg{QueueNotify}{Queue notifier}{0x050}{W}{% - Writing a value to this register notifies the device that - there are new buffers to process in a queue. + \mmioreg{QueueNotify}{Queue notifier}{0x050}{RW}{% + When VIRTIO_F_MMIO_NOTIFICATION has not been negotiated, writing to this + register notifies the device that there are new buffers to process in a queue. - When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, - the value written is the queue index. + When VIRTIO_F_MMIO_NOTIFICATION has been negotiated, reading this register + returns the virtqueue notification structure for calculating notification location. - When VIRTIO_F_NOTIFICATION_DATA has been negotiated, - the \field{Notification data} value has the following format: + See \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notification Structure Layout} + for the notification structure format. - \lstinputlisting{notifications-le.c} - - See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications} - for the definition of the components. + See \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Available Buffer Notifications} + for the notification data format. } \hline \mmioreg{InterruptStatus}{Interrupt status}{0x60}{R}{% @@ -1858,6 +1856,31 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti Further initialization MUST follow the procedure described in \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}. +\subsubsection{Notification Structure Layout}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notification Structure Layout} + +When VIRTIO_F_MMIO_NOTIFICATION has been negotiated, the notification location is calculated +by notification structure. Driver reads \field{QueueNotify} to get this structure formatted +as follows. + +\begin{lstlisting} +le32 { + notify_base : 16; + notify_multiplier : 16; +}; +\end{lstlisting} + +\field{notify_multiplier} is combined with virtqueue index to derive the Queue Notify address +within a memory mapped control registers for a virtqueue: + +\begin{lstlisting} + notify_base + queue_index * notify_multiplier +\end{lstlisting} + +\begin{note} +For example, if notify_multiplier is 0, the device uses the same Queue Notify address for all +queues. +\end{note} + \subsubsection{Virtqueue Configuration}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Virtqueue Configuration} The driver will typically initialize the virtual queue in the following way: @@ -1893,16 +1916,20 @@ \subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Option When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, the driver sends an available buffer notification to the device by writing the 16-bit virtqueue index -of the queue to be notified to \field{QueueNotify}. +of the queue to be notified to Queue Notify address. When VIRTIO_F_NOTIFICATION_DATA has been negotiated, the driver sends an available buffer notification to the device by writing -the following 32-bit value to \field{QueueNotify}: +the following 32-bit value to Queue Notify address: \lstinputlisting{notifications-le.c} See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications} for the definition of the components. +For device not offering VIRTIO_F_MMIO_NOTIFICATION, the Queue Notify address is \field{QueueNotify}. +For device offering VIRTIO_F_MMIO_NOTIFICATION, see \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notification Structure Layout} +for how to calculate the Queue Notify address. + \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device} The memory mapped virtio device is using a single, dedicated From patchwork Tue Jan 21 13:54:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liu, Jing2" X-Patchwork-Id: 1226255 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.intel.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 481z1Q4vp4z9sRl for ; Tue, 21 Jan 2020 17:20:42 +1100 (AEDT) Received: from localhost ([::1]:48924 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmu0-0007L4-3y for incoming@patchwork.ozlabs.org; Tue, 21 Jan 2020 01:20:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41969) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmtE-0007Ho-9Q for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1itmtD-0006Tm-6g for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:52 -0500 Received: from mga03.intel.com ([134.134.136.65]:35723) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1itmtC-0006Rf-UP for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:51 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jan 2020 22:19:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,344,1574150400"; d="scan'208";a="278302031" Received: from hyperv-sh3.bj.intel.com ([10.240.193.95]) by fmsmga001.fm.intel.com with ESMTP; 20 Jan 2020 22:19:49 -0800 From: Jing Liu To: virtio-dev@lists.oasis-open.org Subject: [virtio-dev] [PATCH v2 3/5] virtio-mmio: Add feature bit for MMIO MSI Date: Tue, 21 Jan 2020 21:54:31 +0800 Message-Id: <1579614873-21907-4-git-send-email-jing2.liu@linux.intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> References: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 134.134.136.65 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zha Bin , kvm@vger.kernel.org, Jing Liu , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Chao Peng , Liu Jiang Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The current MMIO transport layer uses a single, dedicated interrupt signal, which brings performance penalty. Add a feature bit (40) for introducing MSI capability. Co-developed-by: Chao Peng Signed-off-by: Chao Peng Co-developed-by: Liu Jiang Signed-off-by: Liu Jiang Co-developed-by: Zha Bin Signed-off-by: Zha Bin Signed-off-by: Jing Liu --- content.tex | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content.tex b/content.tex index 5881253..ff151ba 100644 --- a/content.tex +++ b/content.tex @@ -5840,6 +5840,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_MMIO_NOTIFICATION(39)] This feature indicates that the device supports enhanced notification mechanism on MMIO transport layer. + \item[VIRTIO_F_MMIO_MSI(40)] This feature indicates that the + device supports Message Signal Interrupts (MSI) mechanism on + MMIO transport layer. \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} @@ -5875,6 +5878,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} A driver SHOULD accept VIRTIO_F_MMIO_NOTIFICATION if it is offered. +A driver SHOULD accept VIRTIO_F_MMIO_MSI if it is offered. +If VIRTIO_F_MMIO_MSI has been negotiated, a driver MUST try to +set up MSI at first priority. + \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} A device MUST offer VIRTIO_F_VERSION_1. A device MAY fail to operate further From patchwork Tue Jan 21 13:54:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liu, Jing2" X-Patchwork-Id: 1226259 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.intel.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 481z4t1VZ4z9sRK for ; Tue, 21 Jan 2020 17:23:42 +1100 (AEDT) Received: from localhost ([::1]:48970 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmwu-0003Vq-1w for incoming@patchwork.ozlabs.org; Tue, 21 Jan 2020 01:23:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41986) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmtH-0007ND-Iy for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1itmtF-0006V8-10 for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:55 -0500 Received: from mga03.intel.com ([134.134.136.65]:35723) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1itmtE-0006Rf-KH for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:52 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jan 2020 22:19:52 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,344,1574150400"; d="scan'208";a="278302104" Received: from hyperv-sh3.bj.intel.com ([10.240.193.95]) by fmsmga001.fm.intel.com with ESMTP; 20 Jan 2020 22:19:50 -0800 From: Jing Liu To: virtio-dev@lists.oasis-open.org Subject: [virtio-dev] [PATCH v2 4/5] virtio-mmio: Introduce MSI details Date: Tue, 21 Jan 2020 21:54:32 +0800 Message-Id: <1579614873-21907-5-git-send-email-jing2.liu@linux.intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> References: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 134.134.136.65 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zha Bin , kvm@vger.kernel.org, Jing Liu , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Chao Peng , Liu Jiang Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" With VIRTIO_F_MMIO_MSI feature bit offered, the Message Signal Interrupts (MSI) is supported as first priority. For any reason it fails to use MSI, it need use the single dedicated interrupt as before. For MSI vectors and events mapping relationship, introduce in next patch. Co-developed-by: Chao Peng Signed-off-by: Chao Peng Co-developed-by: Liu Jiang Signed-off-by: Liu Jiang Co-developed-by: Zha Bin Signed-off-by: Zha Bin Signed-off-by: Jing Liu --- content.tex | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++++------ msi-state.c | 4 ++ 2 files changed, 159 insertions(+), 16 deletions(-) create mode 100644 msi-state.c diff --git a/content.tex b/content.tex index ff151ba..dcf6c71 100644 --- a/content.tex +++ b/content.tex @@ -1687,7 +1687,8 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi \hline \mmioreg{InterruptStatus}{Interrupt status}{0x60}{R}{% Reading from this register returns a bit mask of events that - caused the device interrupt to be asserted. + caused the device interrupt to be asserted. This is only used + when MSI is not enabled. The following events are possible: \begin{description} \item[Used Buffer Notification] - bit 0 - the interrupt was asserted @@ -1701,7 +1702,7 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi \mmioreg{InterruptACK}{Interrupt acknowledge}{0x064}{W}{% Writing a value with bits set as defined in \field{InterruptStatus} to this register notifies the device that events causing - the interrupt have been handled. + the interrupt have been handled. This is only used when MSI is not enabled. } \hline \mmioreg{Status}{Device status}{0x070}{RW}{% @@ -1760,6 +1761,47 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi \field{SHMSel} is unused) results in a base address of 0xffffffffffffffff. } + \hline + \mmioreg{MsiVecNum}{MSI max vector number}{0x0c0}{R}{% + When VIRTIO_F_MMIO_MSI has been negotiated, reading + from this register returns the maximum MSI vector number + that device supports. + } + \hline + \mmioreg{MsiState}{MSI state}{0x0c4}{R}{% + When VIRTIO_F_MMIO_MSI has been negotiated, reading + from this register returns the global MSI enable/disable status. + \lstinputlisting{msi-state.c} + } + \hline + \mmioreg{MsiCmd}{MSI command}{0x0c8}{W}{% + When VIRTIO_F_MMIO_MSI has been negotiated, writing + to this register executes the corresponding command to device. + Part of this applies to the MSI vector selected by writing to \field{MsiVecSel}. + See \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization / MSI Vector Configuration} + for using details. + } + \hline + \mmioreg{MsiVecSel}{MSI vector index}{0x0d0}{W}{% + When VIRTIO_F_MMIO_MSI has been negotiated, writing + to this register selects the MSI vector index that the following operations + on \field{MsiAddrLow}, \field{MsiAddrHigh}, \field{MsiData} and part of + \field{MsiCmd} commands specified in \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization / MSI Vector Configuration} + apply to. The index number of the first vector is zero (0x0). + } + \hline + \mmiodreg{MsiAddrLow}{MsiAddrHigh}{MSI 64 bit address}{0x0d4}{0x0d8}{W}{% + When VIRTIO_F_MMIO_MSI has been negotiated, writing + to these two registers (lower 32 bits of the address to \field{MsiAddrLow}, + higher 32 bits to \field{MsiAddrHigh}) notifies the device about the + MSI address. This applies to the MSI vector selected by writing to \field{MsiVecSel}. + } + \hline + \mmioreg{MsiData}{MSI 32 bit data}{0x0dc}{W}{% + When VIRTIO_F_MMIO_MSI has been negotiated, writing + to this register notifies the device about the MSI data. + This applies to the MSI vector selected by writing to \field{MsiVecSel}. + } \hline \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{ Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}). @@ -1783,10 +1825,16 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi The device MUST return value 0x2 in \field{Version}. -The device MUST present each event by setting the corresponding bit in \field{InterruptStatus} from the +When MSI is disabled, the device MUST present each event by setting the +corresponding bit in \field{InterruptStatus} from the moment it takes place, until the driver acknowledges the interrupt -by writing a corresponding bit mask to the \field{InterruptACK} register. Bits which -do not represent events which took place MUST be zero. +by writing a corresponding bit mask to the \field{InterruptACK} register. +Bits which do not represent events which took place MUST be zero. + +When MSI is enabled, the device MUST NOT set \field{InterruptStatus} and MUST +ignore \field{InterruptACK}. + +Upon reset, the device MUST clear \field{msi_enabled} bit in \field{MsiState}. Upon reset, the device MUST clear all bits in \field{InterruptStatus} and ready bits in the \field{QueueReady} register for all queues in the device. @@ -1835,7 +1883,12 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi The driver MUST ignore undefined bits in \field{InterruptStatus}. -The driver MUST write a value with a bit mask describing events it handled into \field{InterruptACK} when +The driver MUST ignore undefined bits in the return value of reading \field{MsiState}. + +When MSI is enabled, the driver MUST NOT access \field{InterruptStatus} and MUST NOT write to \field{InterruptACK}. + +When MSI is disabled, the driver MUST write a value with a bit mask +describing events it handled into \field{InterruptACK} when it finishes handling an interrupt and MUST NOT set any of the undefined bits in the value. \subsection{MMIO-specific Initialization And Device Operation}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation} @@ -1856,6 +1909,63 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti Further initialization MUST follow the procedure described in \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}. +\paragraph{MSI Vector Configuration}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization / MSI Vector Configuration} +The VIRTIO_F_MMIO_MSI feature bit offered by device shows the capability +using MSI vectors for virtqueue and configuration events. + +When VIRTIO_F_MMIO_MSI has been negotiated, +writing \field{MsiCmd} executes a corresponding command to the device: + +VIRTIO_MMIO_MSI_CMD_ENABLE and VIRTIO_MMIO_MSI_CMD_DISABLE commands set global +MSI enable and disable status. + +VIRTIO_MMIO_MSI_CMD_CONFIGURE is used to configure the MSI vector +applying to the one selected by writing to \field{MsiVecSel}. + +VIRTIO_MMIO_MSI_CMD_MASK and VIRTIO_MMIO_MSI_CMD_UNMASK commands are used to +mask and unmask the MSI vector applying to the one selected by writing +to \field{MsiVecSel}. + +\begin{lstlisting} +#define VIRTIO_MMIO_MSI_CMD_ENABLE 0x1 +#define VIRTIO_MMIO_MSI_CMD_DISABLE 0x2 +#define VIRTIO_MMIO_MSI_CMD_CONFIGURE 0x3 +#define VIRTIO_MMIO_MSI_CMD_MASK 0x4 +#define VIRTIO_MMIO_MSI_CMD_UNMASK 0x5 +\end{lstlisting} + +Setting a special NO_VECTOR value means disabling an interrupt for an event type. + +\begin{lstlisting} +/* Vector value used to disable MSI for event */ +#define VIRTIO_MMIO_MSI_NO_VECTOR 0xffffffff +\end{lstlisting} + +\drivernormative{\subparagraph}{MSI Vector Configuration}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / MSI Vector Configuration} +When VIRTIO_F_MMIO_MSI has been negotiated, driver should try to configure +and enable MSI. + +To configure MSI vector, driver SHOULD firstly specify the MSI vector index by +writing to \field{MsiVecSel}. +Then notify the MSI address and data by writing to \field{MsiAddrLow}, \field{MsiAddrHigh}, +and \field{MsiData}, and immediately follow a \field{MsiCmd} write operation +using VIRTIO_MMIO_MSI_CMD_CONFIGURE to device for configuring an event to +this MSI vector. + +After all MSI vectors are configured, driver SHOULD set global MSI enabled +by writing to \field{MsiCmd} using VIRTIO_MMIO_MSI_CMD_ENABLE. + +Driver should use VIRTIO_MMIO_MSI_CMD_DISABLE when disabling MSI. + +Driver should use VIRTIO_MMIO_MSI_CMD_MASK with an MSI index \field{MsiVecSel} +to prohibit the event from the corresponding interrupt source. + +Driver should use VIRTIO_MMIO_MSI_CMD_UNMASK with an MSI index \field{MsiVecSel} +to recover the event from the corresponding interrupt source. + +If driver fails to setup any event with a vector, +it MUST disable MSI by \field{MsiCmd} and use the single dedicated interrupt for device. + \subsubsection{Notification Structure Layout}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notification Structure Layout} When VIRTIO_F_MMIO_NOTIFICATION has been negotiated, the notification location is calculated @@ -1908,6 +2018,12 @@ \subsubsection{Virtqueue Configuration}\label{sec:Virtio Transport Options / Vir \field{QueueDriverLow}/\field{QueueDriverHigh} and \field{QueueDeviceLow}/\field{QueueDeviceHigh} register pairs. +\item Write MSI address \field{MsiAddrLow}/\field{MsiAddrHigh}, +MSI data \field{MsiData} and MSI update command \field{MsiCtrlStat} with corresponding +virtqueue index to update +MSI configuration for device requesting interrupts triggered by +virtqueue events. + \item Write 0x1 to \field{QueueReady}. \end{enumerate} @@ -1932,20 +2048,43 @@ \subsubsection{Available Buffer Notifications}\label{sec:Virtio Transport Option \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device} -The memory mapped virtio device is using a single, dedicated +If MSI is enabled, the memory mapped virtio +device uses appropriate MSI interrupt message +for configuration change notification and used buffer notification which are +configured by \field{MsiAddrLow}, \field{MsoAddrHigh} and \field{MsiData}. + +If MSI is not enabled, the memory mapped virtio device +uses a single, dedicated interrupt signal, which is asserted when at least one of the bits described in the description of \field{InterruptStatus} -is set. This is how the device sends a used buffer notification -or a configuration change notification to the device. +is set. \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device} -After receiving an interrupt, the driver MUST read -\field{InterruptStatus} to check what caused the interrupt (see the -register description). The used buffer notification bit being set -SHOULD be interpreted as a used buffer notification for each active -virtqueue. After the interrupt is handled, the driver MUST acknowledge -it by writing a bit mask corresponding to the handled events to the -InterruptACK register. +A driver MUST handle the case where MSI is disabled, which uses the same interrupt indicating both device configuration +space change and one or more virtqueues being used. + +\subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Driver Handling Interrupts} + +The driver interrupt handler would typically: + +\begin{itemize} + \item If MSI is enabled: + \begin{itemize} + \item + Figure out the virtqueue mapped to that MSI vector for the + device, to see if any progress has been made by the device + which requires servicing. + \item + If the interrupt belongs to configuration space changing signal, + re-examine the configuration space to see what changed. + \end{itemize} + \item If MSI is disabled: + \begin{itemize} + \item Read \field{InterruptStatus} to check what caused the interrupt. + \item Acknowledge the interrupt by writing a bit mask corresponding + to the handled events to the InterruptACK register. + \end{itemize} +\end{itemize} \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface} diff --git a/msi-state.c b/msi-state.c new file mode 100644 index 0000000..b1fa0c1 --- /dev/null +++ b/msi-state.c @@ -0,0 +1,4 @@ +le32 { + msi_enabled : 1; + reserved : 31; +}; From patchwork Tue Jan 21 13:54:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liu, Jing2" X-Patchwork-Id: 1226258 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.intel.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 481z3W10gcz9sP3 for ; Tue, 21 Jan 2020 17:22:31 +1100 (AEDT) Received: from localhost ([::1]:48956 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmvk-0001m7-Pb for incoming@patchwork.ozlabs.org; Tue, 21 Jan 2020 01:22:28 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41991) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1itmtI-0007O8-1A for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1itmtG-0006W1-HV for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:55 -0500 Received: from mga03.intel.com ([134.134.136.65]:35723) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1itmtG-0006Rf-7T for qemu-devel@nongnu.org; Tue, 21 Jan 2020 01:19:54 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jan 2020 22:19:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,344,1574150400"; d="scan'208";a="278302177" Received: from hyperv-sh3.bj.intel.com ([10.240.193.95]) by fmsmga001.fm.intel.com with ESMTP; 20 Jan 2020 22:19:52 -0800 From: Jing Liu To: virtio-dev@lists.oasis-open.org Subject: [virtio-dev] [PATCH v2 5/5] virtio-mmio: MSI vector and event mapping Date: Tue, 21 Jan 2020 21:54:33 +0800 Message-Id: <1579614873-21907-6-git-send-email-jing2.liu@linux.intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> References: <1579614873-21907-1-git-send-email-jing2.liu@linux.intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 134.134.136.65 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zha Bin , kvm@vger.kernel.org, Jing Liu , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Chao Peng , Liu Jiang Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Bit 1 msi_sharing reported in the MsiState register indicates the mapping mode device uses. Bit 1 is 0 - device uses MSI non-sharing mode. This indicates vector per event and fixed static vectors and events relationship. This fits for devices with a high interrupt rate and best performance; Bit 1 is 1 - device uses MSI sharing mode. This indicates vectors and events dynamic mapping and fits for devices not requiring a high interrupt rate. Co-developed-by: Chao Peng Signed-off-by: Chao Peng Co-developed-by: Liu Jiang Signed-off-by: Liu Jiang Co-developed-by: Zha Bin Signed-off-by: Zha Bin Signed-off-by: Jing Liu --- content.tex | 48 +++++++++++++++++++++++++++++++++++++++++++++++- msi-state.c | 3 ++- 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/content.tex b/content.tex index dcf6c71..2fd1686 100644 --- a/content.tex +++ b/content.tex @@ -1770,7 +1770,8 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi \hline \mmioreg{MsiState}{MSI state}{0x0c4}{R}{% When VIRTIO_F_MMIO_MSI has been negotiated, reading - from this register returns the global MSI enable/disable status. + from this register returns the global MSI enable/disable status + and whether device uses MSI sharing mode. \lstinputlisting{msi-state.c} } \hline @@ -1926,12 +1927,18 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti mask and unmask the MSI vector applying to the one selected by writing to \field{MsiVecSel}. +VIRTIO_MMIO_MSI_CMD_MAP_CONFIG command is to set the configuration event and MSI vector +mapping. VIRTIO_MMIO_MSI_CMD_MAP_QUEUE is to set the queue event and MSI vector +mapping. They SHOULD only be used in MSI sharing mode. + \begin{lstlisting} #define VIRTIO_MMIO_MSI_CMD_ENABLE 0x1 #define VIRTIO_MMIO_MSI_CMD_DISABLE 0x2 #define VIRTIO_MMIO_MSI_CMD_CONFIGURE 0x3 #define VIRTIO_MMIO_MSI_CMD_MASK 0x4 #define VIRTIO_MMIO_MSI_CMD_UNMASK 0x5 +#define VIRTIO_MMIO_MSI_CMD_MAP_CONFIG 0x6 +#define VIRTIO_MMIO_MSI_CMD_MAP_QUEUE 0x7 \end{lstlisting} Setting a special NO_VECTOR value means disabling an interrupt for an event type. @@ -1941,10 +1948,49 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti #define VIRTIO_MMIO_MSI_NO_VECTOR 0xffffffff \end{lstlisting} +\subparagraph{MSI Vector and Event Mapping}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization / MSI Vector Configuration} +The reported \field{msi_sharing} bit in the \field{MsiState} return value shows +the MSI sharing mode that device uses. + +When \field{msi_sharing} bit is 0, it indicates the device uses non-sharing mode +and vector per event fixed static relationship is used. The first vector is for device +configuraiton change event, the second vector is for virtqueue 1, the third vector +is for virtqueue 2 and so on. + +When \field{msi_sharing} bit is 1, it indicates the device uses MSI sharing mode, +and the vector and event mapping is dynamic. Writing \field{MsiVecSel} +followed by writing VIRTIO_MMIO_MSI_CMD_MAP_CONFIG/VIRTIO_MMIO_MSI_CMD_MAP_QUEUE command +maps interrupts triggered by the configuration change/selected queue events respectively +to the corresponding MSI vector. + +\devicenormative{\subparagraph}{MSI Vector Configuration}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / MSI Vector Configuration} + +When the device reports \field{msi_sharing} bit as 0, it SHOULD support a number of +vectors that greater than the maximum number of virtqueues. +Device MUST report the number of vectors supported in \field{MsiVecNum}. + +When the device reports \field{msi_sharing} bit as 1, it SHOULD support at least +2 MSI vectors and MUST report in \field{MsiVecNum}. Device SHOULD support mapping any +event type to any vector under \field{MsiVecNum}. + +Device MUST support unmapping any event type (NO_VECTOR). + +The device SHOULD restrict the reported \field{msi_sharing} and \field{MsiVecNum} +to a value that might benefit system performance. + +\begin{note} +For example, a device which does not expect to send interrupts at a high rate might +return \field{msi_sharing} bit as 1. +\end{note} + \drivernormative{\subparagraph}{MSI Vector Configuration}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / MSI Vector Configuration} When VIRTIO_F_MMIO_MSI has been negotiated, driver should try to configure and enable MSI. +To set up the event and vector mapping for MSI sharing mode, driver SHOULD +write a valid \field{MsiVecSel} followed by VIRTIO_MMIO_MSI_CMD_MAP_CONFIG/VIRTIO_MMIO_MSI_CMD_MAP_QUEUE +command to map the configuration change/selected queue events respectively. + To configure MSI vector, driver SHOULD firstly specify the MSI vector index by writing to \field{MsiVecSel}. Then notify the MSI address and data by writing to \field{MsiAddrLow}, \field{MsiAddrHigh}, diff --git a/msi-state.c b/msi-state.c index b1fa0c1..d470be4 100644 --- a/msi-state.c +++ b/msi-state.c @@ -1,4 +1,5 @@ le32 { msi_enabled : 1; - reserved : 31; + msi_sharing: 1; + reserved : 30; };