[RESEND,v9,4/5] ndctl/papr_scm, uapi: Add support for PAPR nvdimm specific methods

Introduce support for PAPR NVDIMM Specific Methods (PDSM) in papr_scm
module and add the command family NVDIMM_FAMILY_PAPR to the white list
of NVDIMM command sets. Also advertise support for ND_CMD_CALL for the
nvdimm command mask and implement necessary scaffolding in the module
to handle ND_CMD_CALL ioctl and PDSM requests that we receive.

The layout of the PDSM request as we expect from libnvdimm/libndctl is
described in newly introduced uapi header 'papr_pdsm.h' which
defines a new 'struct nd_pdsm_cmd_pkg' header. This header is used
to communicate the PDSM request via member
'nd_cmd_pkg.nd_command' and size of payload that need to be
sent/received for servicing the PDSM.

A new function is_cmd_valid() is implemented that reads the args to
papr_scm_ndctl() and performs sanity tests on them. A new function
papr_scm_service_pdsm() is introduced and is called from
papr_scm_ndctl() in case of a PDSM request is received via ND_CMD_CALL
command from libnvdimm.

Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Ira Weiny <ira.weiny@intel.com>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
---
Changelog:

Resend:
* Added ack from Aneesh.

v8..v9:
* Reduced the usage of term SCM replacing it with appropriate
  replacement [ Dan Williams, Aneesh ]
* Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
* s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
* s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
* Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
* Minor update to patch description.

v7..v8:
* Removed the 'payload_offset' field from 'struct
  nd_pdsm_cmd_pkg'. Instead command payload is always assumed to start
  at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
* To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
  'reserved' field of 10-bytes is introduced. [ Aneesh ]
* Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
  [ Ira ]

Resend:
* None

v6..v7 :
* Removed the re-definitions of __packed macro from papr_scm_pdsm.h
  [Mpe].
* Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h [Mpe].
* Removed macros that were unused in papr_scm.c from papr_scm_pdsm.h
  [Mpe].
* Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]

v5..v6 :
* Changed the usage of the term DSM to PDSM to distinguish it from the
  ACPI term [ Dan Williams ]
* Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various struct
  to reflect the new terminology.
* Updated the patch description and title to reflect the new terminology.
* Squashed patch to introduce new command family in 'ndctl.h' with
  this patch [ Dan Williams ]
* Updated the papr_scm_pdsm method starting index from 0x10000 to 0x0
  [ Dan Williams ]
* Removed redundant license text from the papr_scm_psdm.h file.
  [ Dan Williams ]
* s/envelop/envelope/ at various places [ Dan Williams ]
* Added '__packed' attribute to command package header to gaurd
  against different compiler adding paddings between the fields.
  [ Dan Williams]
* Converted various pr_debug to dev_debug [ Dan Williams ]

v4..v5 :
* None

v3..v4 :
* None

v2..v3 :
* Updated the patch prefix to 'ndctl/uapi' [Aneesh]

v1..v2 :
* None
---
 arch/powerpc/include/uapi/asm/papr_pdsm.h | 136 ++++++++++++++++++++++
 arch/powerpc/platforms/pseries/papr_scm.c | 101 +++++++++++++++-
 include/uapi/linux/ndctl.h                |   1 +
 3 files changed, 232 insertions(+), 6 deletions(-)
 create mode 100644 arch/powerpc/include/uapi/asm/papr_pdsm.h

On Tue, Jun 02, 2020 at 03:44:37PM +0530, Vaibhav Jain wrote:
> Introduce support for PAPR NVDIMM Specific Methods (PDSM) in papr_scm
> module and add the command family NVDIMM_FAMILY_PAPR to the white list
> of NVDIMM command sets. Also advertise support for ND_CMD_CALL for the
> nvdimm command mask and implement necessary scaffolding in the module
> to handle ND_CMD_CALL ioctl and PDSM requests that we receive.
> 
> The layout of the PDSM request as we expect from libnvdimm/libndctl is
> described in newly introduced uapi header 'papr_pdsm.h' which
> defines a new 'struct nd_pdsm_cmd_pkg' header. This header is used
> to communicate the PDSM request via member
> 'nd_cmd_pkg.nd_command' and size of payload that need to be
> sent/received for servicing the PDSM.
> 
> A new function is_cmd_valid() is implemented that reads the args to
> papr_scm_ndctl() and performs sanity tests on them. A new function
> papr_scm_service_pdsm() is introduced and is called from
> papr_scm_ndctl() in case of a PDSM request is received via ND_CMD_CALL
> command from libnvdimm.
> 
> Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Ira Weiny <ira.weiny@intel.com>
> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
> Changelog:
> 
> Resend:
> * Added ack from Aneesh.
> 
> v8..v9:
> * Reduced the usage of term SCM replacing it with appropriate
>   replacement [ Dan Williams, Aneesh ]
> * Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
> * s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
> * s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
> * Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
> * Minor update to patch description.
> 
> v7..v8:
> * Removed the 'payload_offset' field from 'struct
>   nd_pdsm_cmd_pkg'. Instead command payload is always assumed to start
>   at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
> * To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
>   'reserved' field of 10-bytes is introduced. [ Aneesh ]
> * Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
>   [ Ira ]
> 
> Resend:
> * None
> 
> v6..v7 :
> * Removed the re-definitions of __packed macro from papr_scm_pdsm.h
>   [Mpe].
> * Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h [Mpe].
> * Removed macros that were unused in papr_scm.c from papr_scm_pdsm.h
>   [Mpe].
> * Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]
> 
> v5..v6 :
> * Changed the usage of the term DSM to PDSM to distinguish it from the
>   ACPI term [ Dan Williams ]
> * Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various struct
>   to reflect the new terminology.
> * Updated the patch description and title to reflect the new terminology.
> * Squashed patch to introduce new command family in 'ndctl.h' with
>   this patch [ Dan Williams ]
> * Updated the papr_scm_pdsm method starting index from 0x10000 to 0x0
>   [ Dan Williams ]
> * Removed redundant license text from the papr_scm_psdm.h file.
>   [ Dan Williams ]
> * s/envelop/envelope/ at various places [ Dan Williams ]
> * Added '__packed' attribute to command package header to gaurd
>   against different compiler adding paddings between the fields.
>   [ Dan Williams]
> * Converted various pr_debug to dev_debug [ Dan Williams ]
> 
> v4..v5 :
> * None
> 
> v3..v4 :
> * None
> 
> v2..v3 :
> * Updated the patch prefix to 'ndctl/uapi' [Aneesh]
> 
> v1..v2 :
> * None
> ---
>  arch/powerpc/include/uapi/asm/papr_pdsm.h | 136 ++++++++++++++++++++++
>  arch/powerpc/platforms/pseries/papr_scm.c | 101 +++++++++++++++-
>  include/uapi/linux/ndctl.h                |   1 +
>  3 files changed, 232 insertions(+), 6 deletions(-)
>  create mode 100644 arch/powerpc/include/uapi/asm/papr_pdsm.h
> 
> diff --git a/arch/powerpc/include/uapi/asm/papr_pdsm.h b/arch/powerpc/include/uapi/asm/papr_pdsm.h
> new file mode 100644
> index 000000000000..6407fefcc007
> --- /dev/null
> +++ b/arch/powerpc/include/uapi/asm/papr_pdsm.h
> @@ -0,0 +1,136 @@
> +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
> +/*
> + * PAPR nvDimm Specific Methods (PDSM) and structs for libndctl
> + *
> + * (C) Copyright IBM 2020
> + *
> + * Author: Vaibhav Jain <vaibhav at linux.ibm.com>
> + */
> +
> +#ifndef _UAPI_ASM_POWERPC_PAPR_PDSM_H_
> +#define _UAPI_ASM_POWERPC_PAPR_PDSM_H_
> +
> +#include <linux/types.h>
> +
> +/*
> + * PDSM Envelope:
> + *
> + * The ioctl ND_CMD_CALL transfers data between user-space and kernel via
> + * envelope which consists of a header and user-defined payload sections.
> + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
> + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
> + * There is reserved field that can used to introduce new fields to the
> + * structure in future. It also tries to ensure that 'nd_pdsm_cmd_pkg.payload'
> + * lies at a 8-byte boundary.
> + *
> + *  +-------------+---------------------+---------------------------+
> + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
> + *  +-------------+---------------------+---------------------------+
> + *  |               nd_pdsm_cmd_pkg     |                           |
> + *  |-------------+                     |                           |
> + *  |  nd_cmd_pkg |                     |                           |
> + *  +-------------+---------------------+---------------------------+
> + *  | nd_family   |                     |                           |
> + *  | nd_size_out | cmd_status          |                           |
> + *  | nd_size_in  | payload_version     |     payload               |
> + *  | nd_command  | reserved            |                           |
> + *  | nd_fw_size  |                     |                           |
> + *  +-------------+---------------------+---------------------------+
> + *
> + * PDSM Header:
> + *
> + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
> + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to member
> + * 'nd_cmd_pkg.nd_command'. Apart from size information of the envelope which is
> + * contained in 'struct nd_cmd_pkg', the header also has members following 
                                                             ^^^^^
                                                        ...  the  ...

> + * members:
> + *
> + * 'cmd_status'		: (Out) Errors if any encountered while servicing PDSM.
> + * 'payload_version'	: (In/Out) Version number associated with the payload.
> + * 'reserved'		: Not used and reserved for future.
> + *
> + * PDSM Payload:
> + *
> + * The layout of the PDSM Payload is defined by various structs shared between
> + * papr_scm and libndctl so that contents of payload can be interpreted. During
> + * servicing of a PDSM the papr_scm module will read input args from the payload
> + * field by casting its contents to an appropriate struct pointer based on the
> + * PDSM command. Similarly the output of servicing the PDSM command will be
> + * copied to the payload field using the same struct.
> + *
> + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size, which
> + * leaves around 176 bytes for the envelope payload (ignoring any padding that
> + * the compiler may silently introduce).
> + *
> + * Payload Version:
> + *
> + * A 'payload_version' field is present in PDSM header that indicates a specific
> + * version of the structure present in PDSM Payload for a given PDSM command.
> + * This provides backward compatibility in case the PDSM Payload structure
> + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
> + *
> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
> + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
> + * module when servicing the PDSM envelope checks the 'payload_version' and then
> + * uses 'payload struct version' == MIN('payload_version field',
> + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
> + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
> + * struct in returned 'payload_version' field.
> + *
> + * Libndctl on receiving the envelope back from papr_scm again checks the
> + * 'payload_version' field and based on it use the appropriate version dsm
> + * struct to parse the results.
> + *
> + * Backward Compatibility:
> + *
> + * Above scheme of exchanging different versioned PDSM struct between libndctl
> + * and papr_scm should provide backward compatibility until following two
> + * assumptions/conditions when defining new PDSM structs hold:
> + *
> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
> + *
> + * 1. T(X) is a proper subset of T(Y) if Y > X.
> + *    i.e Each new version of PDSM struct should retain existing struct
> + *    attributes from previous version
> + *
> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
> + *    it should also support T(1), T(2)...T(X - 1).
> + *    i.e When adding support for new version of a PDSM struct, libndctl
> + *    and papr_scm should retain support of the existing PDSM struct
> + *    version they support.

Please see this thread for an example why versions are a bad idea in UAPIs:

https://lkml.org/lkml/2020/3/26/213

While the use of version is different in that thread the fundamental issues are
the same.  You end up with some weird matrix of supported features and
structure definitions.  For example, you are opening up the possibility of
changing structures with a different version for no good reason.

Also having the user query with version Z and get back version X (older) is
odd.  Generally if the kernel does not know about a feature (ie version Z of
the structure) it should return -EINVAL and let the user figure out what to do.
The user may just give up or they could try a different query.

> + */
> +
> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
> +struct nd_pdsm_cmd_pkg {
> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
> +	__u16 reserved[5];	/* Ignored and to be used in future */

How do you know when reserved is used for something else in the future?  Is
reserved guaranteed (and checked by the code) to be 0?

> +	__u16 payload_version;	/* In/Out: version of the payload */

Why is payload_version after reserved?

> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
> +} __packed;
> +
> +/*
> + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
> + */
> +enum papr_pdsm {
> +	PAPR_PDSM_MIN = 0x0,
> +	PAPR_PDSM_MAX,
> +};
> +
> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
> +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
> +{
> +	return (struct nd_pdsm_cmd_pkg *) cmd;
> +}
> +
> +/* Return the payload pointer for a given pcmd */
> +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
> +{
> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
> +		return NULL;
> +	else
> +		return (void *)(pcmd->payload);
> +}
> +
> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
> index 149431594839..5e2237e7ec08 100644
> --- a/arch/powerpc/platforms/pseries/papr_scm.c
> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> @@ -15,13 +15,15 @@
>  #include <linux/seq_buf.h>
>  
>  #include <asm/plpar_wrappers.h>
> +#include <asm/papr_pdsm.h>
>  
>  #define BIND_ANY_ADDR (~0ul)
>  
>  #define PAPR_SCM_DIMM_CMD_MASK \
>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
> +	 (1ul << ND_CMD_CALL))
>  
>  /* DIMM health bitmap bitmap indicators */
>  /* SCM device is unable to persist memory contents */
> @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>  	return 0;
>  }
>  
> +/*
> + * Validate the inputs args to dimm-control function and return '0' if valid.
> + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
> + */
> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> +		       unsigned int buf_len)
> +{
> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
> +	struct papr_scm_priv *p;
> +
> +	/* Only dimm-specific calls are supported atm */
> +	if (!nvdimm)
> +		return -EINVAL;
> +
> +	/* get the provider date from struct nvdimm */

s/date/data

> +	p = nvdimm_provider_data(nvdimm);
> +
> +	if (!test_bit(cmd, &cmd_mask)) {
> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
> +		return -EINVAL;
> +	} else if (cmd == ND_CMD_CALL) {
> +
> +		/* Verify the envelope package */
> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
> +				buf_len);
> +			return -EINVAL;
> +		}
> +
> +		/* Verify that the PDSM family is valid */
> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
> +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
> +				pkg->hdr.nd_family);
> +			return -EINVAL;
> +
> +		}
> +
> +		/* We except a payload with all PDSM commands */
> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
> +			dev_dbg(&p->pdev->dev,
> +				"Empty payload for sub-command=0x%llx\n",
> +				pkg->hdr.nd_command);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	/* Command looks valid */

I assume the first command to be implemented also checks the { nd_command,
payload_version, payload length } for correctness?

> +	return 0;
> +}
> +
> +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
> +				struct nd_pdsm_cmd_pkg *call_pkg)
> +{
> +	/* unknown subcommands return error in packages */
> +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
> +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
> +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
> +			call_pkg->hdr.nd_command);
> +		call_pkg->cmd_status = -EINVAL;
> +		return 0;
> +	}
> +
> +	/* Depending on the DSM command call appropriate service routine */
> +	switch (call_pkg->hdr.nd_command) {
> +	default:
> +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
> +			call_pkg->hdr.nd_command);
> +		call_pkg->cmd_status = -ENOENT;
> +		return 0;
> +	}
> +}
> +
>  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>  			  unsigned int buf_len, int *cmd_rc)
>  {
>  	struct nd_cmd_get_config_size *get_size_hdr;
>  	struct papr_scm_priv *p;
> +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
> +	int rc;
>  
> -	/* Only dimm-specific calls are supported atm */
> -	if (!nvdimm)
> -		return -EINVAL;
> +	/* Use a local variable in case cmd_rc pointer is NULL */
> +	if (cmd_rc == NULL)
> +		cmd_rc = &rc;

Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
and you did not change it.

> +
> +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
> +	if (*cmd_rc) {
> +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
> +		return *cmd_rc;
> +	}
>  
>  	p = nvdimm_provider_data(nvdimm);
>  
> @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>  		*cmd_rc = papr_scm_meta_set(p, buf);
>  		break;
>  
> +	case ND_CMD_CALL:
> +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
> +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
> +		break;
> +
>  	default:
> -		return -EINVAL;
> +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
> +		*cmd_rc = -EINVAL;

Is this change related?  If there is a bug where there is a caller of
papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
that issue.

Ira

>  	}
>  
>  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
>  
> -	return 0;
> +	return *cmd_rc;
>  }
>  
>  static ssize_t flags_show(struct device *dev,
> diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
> index de5d90212409..0e09dc5cec19 100644
> --- a/include/uapi/linux/ndctl.h
> +++ b/include/uapi/linux/ndctl.h
> @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
>  #define NVDIMM_FAMILY_HPE2 2
>  #define NVDIMM_FAMILY_MSFT 3
>  #define NVDIMM_FAMILY_HYPERV 4
> +#define NVDIMM_FAMILY_PAPR 5
>  
>  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
>  					struct nd_cmd_pkg)
> -- 
> 2.26.2
>

On Tue, Jun 02, 2020 at 01:51:49PM -0700, 'Ira Weiny' wrote:
> On Tue, Jun 02, 2020 at 03:44:37PM +0530, Vaibhav Jain wrote:

...

> > +
> > +/*
> > + * PDSM Envelope:
> > + *
> > + * The ioctl ND_CMD_CALL transfers data between user-space and kernel via
> > + * envelope which consists of a header and user-defined payload sections.
> > + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
> > + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
> > + * There is reserved field that can used to introduce new fields to the
> > + * structure in future. It also tries to ensure that 'nd_pdsm_cmd_pkg.payload'
> > + * lies at a 8-byte boundary.
> > + *
> > + *  +-------------+---------------------+---------------------------+
> > + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
> > + *  +-------------+---------------------+---------------------------+
> > + *  |               nd_pdsm_cmd_pkg     |                           |
> > + *  |-------------+                     |                           |
> > + *  |  nd_cmd_pkg |                     |                           |
> > + *  +-------------+---------------------+---------------------------+
> > + *  | nd_family   |                     |                           |
> > + *  | nd_size_out | cmd_status          |                           |
> > + *  | nd_size_in  | payload_version     |     payload               |
> > + *  | nd_command  | reserved            |                           |
> > + *  | nd_fw_size  |                     |                           |
> > + *  +-------------+---------------------+---------------------------+

One more comment WRT nd_size_[in|out].  I know that it is defined as the size
of the FW payload but normally when you nest headers 'size' in Header A
represents everything after Header A, including Header B.  In this case that
would be including nd_pdsm_cmd_pkg...

It looks like that is not what you have done?  Or perhaps I missed it?

Ira

> > + *
> > + * PDSM Header:
> > + *
> > + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
> > + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to member
> > + * 'nd_cmd_pkg.nd_command'. Apart from size information of the envelope which is
> > + * contained in 'struct nd_cmd_pkg', the header also has members following 
>                                                              ^^^^^
>                                                         ...  the  ...
> 
> > + * members:
> > + *
> > + * 'cmd_status'		: (Out) Errors if any encountered while servicing PDSM.
> > + * 'payload_version'	: (In/Out) Version number associated with the payload.
> > + * 'reserved'		: Not used and reserved for future.
> > + *
> > + * PDSM Payload:
> > + *
> > + * The layout of the PDSM Payload is defined by various structs shared between
> > + * papr_scm and libndctl so that contents of payload can be interpreted. During
> > + * servicing of a PDSM the papr_scm module will read input args from the payload
> > + * field by casting its contents to an appropriate struct pointer based on the
> > + * PDSM command. Similarly the output of servicing the PDSM command will be
> > + * copied to the payload field using the same struct.
> > + *
> > + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size, which
> > + * leaves around 176 bytes for the envelope payload (ignoring any padding that
> > + * the compiler may silently introduce).
> > + *
> > + * Payload Version:
> > + *
> > + * A 'payload_version' field is present in PDSM header that indicates a specific
> > + * version of the structure present in PDSM Payload for a given PDSM command.
> > + * This provides backward compatibility in case the PDSM Payload structure
> > + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
> > + *
> > + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
> > + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
> > + * module when servicing the PDSM envelope checks the 'payload_version' and then
> > + * uses 'payload struct version' == MIN('payload_version field',
> > + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
> > + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
> > + * struct in returned 'payload_version' field.
> > + *
> > + * Libndctl on receiving the envelope back from papr_scm again checks the
> > + * 'payload_version' field and based on it use the appropriate version dsm
> > + * struct to parse the results.
> > + *
> > + * Backward Compatibility:
> > + *
> > + * Above scheme of exchanging different versioned PDSM struct between libndctl
> > + * and papr_scm should provide backward compatibility until following two
> > + * assumptions/conditions when defining new PDSM structs hold:
> > + *
> > + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
> > + *
> > + * 1. T(X) is a proper subset of T(Y) if Y > X.
> > + *    i.e Each new version of PDSM struct should retain existing struct
> > + *    attributes from previous version
> > + *
> > + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
> > + *    it should also support T(1), T(2)...T(X - 1).
> > + *    i.e When adding support for new version of a PDSM struct, libndctl
> > + *    and papr_scm should retain support of the existing PDSM struct
> > + *    version they support.
> 
> Please see this thread for an example why versions are a bad idea in UAPIs:
> 
> https://lkml.org/lkml/2020/3/26/213
> 
> While the use of version is different in that thread the fundamental issues are
> the same.  You end up with some weird matrix of supported features and
> structure definitions.  For example, you are opening up the possibility of
> changing structures with a different version for no good reason.
> 
> Also having the user query with version Z and get back version X (older) is
> odd.  Generally if the kernel does not know about a feature (ie version Z of
> the structure) it should return -EINVAL and let the user figure out what to do.
> The user may just give up or they could try a different query.
> 
> > + */
> > +
> > +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
> > +struct nd_pdsm_cmd_pkg {
> > +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
> > +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
> > +	__u16 reserved[5];	/* Ignored and to be used in future */
> 
> How do you know when reserved is used for something else in the future?  Is
> reserved guaranteed (and checked by the code) to be 0?
> 
> > +	__u16 payload_version;	/* In/Out: version of the payload */
> 
> Why is payload_version after reserved?
> 
> > +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
> > +} __packed;
> > +
> > +/*
> > + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
> > + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
> > + */
> > +enum papr_pdsm {
> > +	PAPR_PDSM_MIN = 0x0,
> > +	PAPR_PDSM_MAX,
> > +};
> > +
> > +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
> > +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
> > +{
> > +	return (struct nd_pdsm_cmd_pkg *) cmd;
> > +}
> > +
> > +/* Return the payload pointer for a given pcmd */
> > +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
> > +{
> > +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
> > +		return NULL;
> > +	else
> > +		return (void *)(pcmd->payload);
> > +}
> > +
> > +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
> > diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
> > index 149431594839..5e2237e7ec08 100644
> > --- a/arch/powerpc/platforms/pseries/papr_scm.c
> > +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> > @@ -15,13 +15,15 @@
> >  #include <linux/seq_buf.h>
> >  
> >  #include <asm/plpar_wrappers.h>
> > +#include <asm/papr_pdsm.h>
> >  
> >  #define BIND_ANY_ADDR (~0ul)
> >  
> >  #define PAPR_SCM_DIMM_CMD_MASK \
> >  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
> >  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
> > -	 (1ul << ND_CMD_SET_CONFIG_DATA))
> > +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
> > +	 (1ul << ND_CMD_CALL))
> >  
> >  /* DIMM health bitmap bitmap indicators */
> >  /* SCM device is unable to persist memory contents */
> > @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
> >  	return 0;
> >  }
> >  
> > +/*
> > + * Validate the inputs args to dimm-control function and return '0' if valid.
> > + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
> > + */
> > +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> > +		       unsigned int buf_len)
> > +{
> > +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
> > +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
> > +	struct papr_scm_priv *p;
> > +
> > +	/* Only dimm-specific calls are supported atm */
> > +	if (!nvdimm)
> > +		return -EINVAL;
> > +
> > +	/* get the provider date from struct nvdimm */
> 
> s/date/data
> 
> > +	p = nvdimm_provider_data(nvdimm);
> > +
> > +	if (!test_bit(cmd, &cmd_mask)) {
> > +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
> > +		return -EINVAL;
> > +	} else if (cmd == ND_CMD_CALL) {
> > +
> > +		/* Verify the envelope package */
> > +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
> > +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
> > +				buf_len);
> > +			return -EINVAL;
> > +		}
> > +
> > +		/* Verify that the PDSM family is valid */
> > +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
> > +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
> > +				pkg->hdr.nd_family);
> > +			return -EINVAL;
> > +
> > +		}
> > +
> > +		/* We except a payload with all PDSM commands */
> > +		if (pdsm_cmd_to_payload(pkg) == NULL) {
> > +			dev_dbg(&p->pdev->dev,
> > +				"Empty payload for sub-command=0x%llx\n",
> > +				pkg->hdr.nd_command);
> > +			return -EINVAL;
> > +		}
> > +	}
> > +
> > +	/* Command looks valid */
> 
> I assume the first command to be implemented also checks the { nd_command,
> payload_version, payload length } for correctness?
> 
> > +	return 0;
> > +}
> > +
> > +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
> > +				struct nd_pdsm_cmd_pkg *call_pkg)
> > +{
> > +	/* unknown subcommands return error in packages */
> > +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
> > +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
> > +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
> > +			call_pkg->hdr.nd_command);
> > +		call_pkg->cmd_status = -EINVAL;
> > +		return 0;
> > +	}
> > +
> > +	/* Depending on the DSM command call appropriate service routine */
> > +	switch (call_pkg->hdr.nd_command) {
> > +	default:
> > +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
> > +			call_pkg->hdr.nd_command);
> > +		call_pkg->cmd_status = -ENOENT;
> > +		return 0;
> > +	}
> > +}
> > +
> >  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
> >  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> >  			  unsigned int buf_len, int *cmd_rc)
> >  {
> >  	struct nd_cmd_get_config_size *get_size_hdr;
> >  	struct papr_scm_priv *p;
> > +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
> > +	int rc;
> >  
> > -	/* Only dimm-specific calls are supported atm */
> > -	if (!nvdimm)
> > -		return -EINVAL;
> > +	/* Use a local variable in case cmd_rc pointer is NULL */
> > +	if (cmd_rc == NULL)
> > +		cmd_rc = &rc;
> 
> Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
> and you did not change it.
> 
> > +
> > +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
> > +	if (*cmd_rc) {
> > +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
> > +		return *cmd_rc;
> > +	}
> >  
> >  	p = nvdimm_provider_data(nvdimm);
> >  
> > @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
> >  		*cmd_rc = papr_scm_meta_set(p, buf);
> >  		break;
> >  
> > +	case ND_CMD_CALL:
> > +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
> > +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
> > +		break;
> > +
> >  	default:
> > -		return -EINVAL;
> > +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
> > +		*cmd_rc = -EINVAL;
> 
> Is this change related?  If there is a bug where there is a caller of
> papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
> that issue.
> 
> Ira
> 
> >  	}
> >  
> >  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
> >  
> > -	return 0;
> > +	return *cmd_rc;
> >  }
> >  
> >  static ssize_t flags_show(struct device *dev,
> > diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
> > index de5d90212409..0e09dc5cec19 100644
> > --- a/include/uapi/linux/ndctl.h
> > +++ b/include/uapi/linux/ndctl.h
> > @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
> >  #define NVDIMM_FAMILY_HPE2 2
> >  #define NVDIMM_FAMILY_MSFT 3
> >  #define NVDIMM_FAMILY_HYPERV 4
> > +#define NVDIMM_FAMILY_PAPR 5
> >  
> >  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
> >  					struct nd_cmd_pkg)
> > -- 
> > 2.26.2
> > 
> _______________________________________________
> Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
> To unsubscribe send an email to linux-nvdimm-leave@lists.01.org

Hi Ira,

Thanks for reviewing this patch. My responses below:

Ira Weiny <ira.weiny@intel.com> writes:

> On Tue, Jun 02, 2020 at 03:44:37PM +0530, Vaibhav Jain wrote:
>> Introduce support for PAPR NVDIMM Specific Methods (PDSM) in papr_scm
>> module and add the command family NVDIMM_FAMILY_PAPR to the white list
>> of NVDIMM command sets. Also advertise support for ND_CMD_CALL for the
>> nvdimm command mask and implement necessary scaffolding in the module
>> to handle ND_CMD_CALL ioctl and PDSM requests that we receive.
>> 
>> The layout of the PDSM request as we expect from libnvdimm/libndctl is
>> described in newly introduced uapi header 'papr_pdsm.h' which
>> defines a new 'struct nd_pdsm_cmd_pkg' header. This header is used
>> to communicate the PDSM request via member
>> 'nd_cmd_pkg.nd_command' and size of payload that need to be
>> sent/received for servicing the PDSM.
>> 
>> A new function is_cmd_valid() is implemented that reads the args to
>> papr_scm_ndctl() and performs sanity tests on them. A new function
>> papr_scm_service_pdsm() is introduced and is called from
>> papr_scm_ndctl() in case of a PDSM request is received via ND_CMD_CALL
>> command from libnvdimm.
>> 
>> Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
>> Cc: Dan Williams <dan.j.williams@intel.com>
>> Cc: Michael Ellerman <mpe@ellerman.id.au>
>> Cc: Ira Weiny <ira.weiny@intel.com>
>> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
>> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
>> ---
>> Changelog:
>> 
>> Resend:
>> * Added ack from Aneesh.
>> 
>> v8..v9:
>> * Reduced the usage of term SCM replacing it with appropriate
>>   replacement [ Dan Williams, Aneesh ]
>> * Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
>> * s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
>> * s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
>> * Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
>> * Minor update to patch description.
>> 
>> v7..v8:
>> * Removed the 'payload_offset' field from 'struct
>>   nd_pdsm_cmd_pkg'. Instead command payload is always assumed to start
>>   at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
>> * To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
>>   'reserved' field of 10-bytes is introduced. [ Aneesh ]
>> * Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
>>   [ Ira ]
>> 
>> Resend:
>> * None
>> 
>> v6..v7 :
>> * Removed the re-definitions of __packed macro from papr_scm_pdsm.h
>>   [Mpe].
>> * Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h [Mpe].
>> * Removed macros that were unused in papr_scm.c from papr_scm_pdsm.h
>>   [Mpe].
>> * Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]
>> 
>> v5..v6 :
>> * Changed the usage of the term DSM to PDSM to distinguish it from the
>>   ACPI term [ Dan Williams ]
>> * Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various struct
>>   to reflect the new terminology.
>> * Updated the patch description and title to reflect the new terminology.
>> * Squashed patch to introduce new command family in 'ndctl.h' with
>>   this patch [ Dan Williams ]
>> * Updated the papr_scm_pdsm method starting index from 0x10000 to 0x0
>>   [ Dan Williams ]
>> * Removed redundant license text from the papr_scm_psdm.h file.
>>   [ Dan Williams ]
>> * s/envelop/envelope/ at various places [ Dan Williams ]
>> * Added '__packed' attribute to command package header to gaurd
>>   against different compiler adding paddings between the fields.
>>   [ Dan Williams]
>> * Converted various pr_debug to dev_debug [ Dan Williams ]
>> 
>> v4..v5 :
>> * None
>> 
>> v3..v4 :
>> * None
>> 
>> v2..v3 :
>> * Updated the patch prefix to 'ndctl/uapi' [Aneesh]
>> 
>> v1..v2 :
>> * None
>> ---
>>  arch/powerpc/include/uapi/asm/papr_pdsm.h | 136 ++++++++++++++++++++++
>>  arch/powerpc/platforms/pseries/papr_scm.c | 101 +++++++++++++++-
>>  include/uapi/linux/ndctl.h                |   1 +
>>  3 files changed, 232 insertions(+), 6 deletions(-)
>>  create mode 100644 arch/powerpc/include/uapi/asm/papr_pdsm.h
>> 
>> diff --git a/arch/powerpc/include/uapi/asm/papr_pdsm.h b/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> new file mode 100644
>> index 000000000000..6407fefcc007
>> --- /dev/null
>> +++ b/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> @@ -0,0 +1,136 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
>> +/*
>> + * PAPR nvDimm Specific Methods (PDSM) and structs for libndctl
>> + *
>> + * (C) Copyright IBM 2020
>> + *
>> + * Author: Vaibhav Jain <vaibhav at linux.ibm.com>
>> + */
>> +
>> +#ifndef _UAPI_ASM_POWERPC_PAPR_PDSM_H_
>> +#define _UAPI_ASM_POWERPC_PAPR_PDSM_H_
>> +
>> +#include <linux/types.h>
>> +
>> +/*
>> + * PDSM Envelope:
>> + *
>> + * The ioctl ND_CMD_CALL transfers data between user-space and kernel via
>> + * envelope which consists of a header and user-defined payload sections.
>> + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
>> + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
>> + * There is reserved field that can used to introduce new fields to the
>> + * structure in future. It also tries to ensure that 'nd_pdsm_cmd_pkg.payload'
>> + * lies at a 8-byte boundary.
>> + *
>> + *  +-------------+---------------------+---------------------------+
>> + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
>> + *  +-------------+---------------------+---------------------------+
>> + *  |               nd_pdsm_cmd_pkg     |                           |
>> + *  |-------------+                     |                           |
>> + *  |  nd_cmd_pkg |                     |                           |
>> + *  +-------------+---------------------+---------------------------+
>> + *  | nd_family   |                     |                           |
>> + *  | nd_size_out | cmd_status          |                           |
>> + *  | nd_size_in  | payload_version     |     payload               |
>> + *  | nd_command  | reserved            |                           |
>> + *  | nd_fw_size  |                     |                           |
>> + *  +-------------+---------------------+---------------------------+
>> + *
>> + * PDSM Header:
>> + *
>> + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
>> + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to member
>> + * 'nd_cmd_pkg.nd_command'. Apart from size information of the envelope which is
>> + * contained in 'struct nd_cmd_pkg', the header also has members following 
>                                                              ^^^^^
>                                                         ...  the  ...
Thanks for catching this will get it fixed.
>
>> + * members:
>> + *
>> + * 'cmd_status'		: (Out) Errors if any encountered while servicing PDSM.
>> + * 'payload_version'	: (In/Out) Version number associated with the payload.
>> + * 'reserved'		: Not used and reserved for future.
>> + *
>> + * PDSM Payload:
>> + *
>> + * The layout of the PDSM Payload is defined by various structs shared between
>> + * papr_scm and libndctl so that contents of payload can be interpreted. During
>> + * servicing of a PDSM the papr_scm module will read input args from the payload
>> + * field by casting its contents to an appropriate struct pointer based on the
>> + * PDSM command. Similarly the output of servicing the PDSM command will be
>> + * copied to the payload field using the same struct.
>> + *
>> + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size, which
>> + * leaves around 176 bytes for the envelope payload (ignoring any padding that
>> + * the compiler may silently introduce).
>> + *
>> + * Payload Version:
>> + *
>> + * A 'payload_version' field is present in PDSM header that indicates a specific
>> + * version of the structure present in PDSM Payload for a given PDSM command.
>> + * This provides backward compatibility in case the PDSM Payload structure
>> + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
>> + *
>> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
>> + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
>> + * module when servicing the PDSM envelope checks the 'payload_version' and then
>> + * uses 'payload struct version' == MIN('payload_version field',
>> + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
>> + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
>> + * struct in returned 'payload_version' field.
>> + *
>> + * Libndctl on receiving the envelope back from papr_scm again checks the
>> + * 'payload_version' field and based on it use the appropriate version dsm
>> + * struct to parse the results.
>> + *
>> + * Backward Compatibility:
>> + *
>> + * Above scheme of exchanging different versioned PDSM struct between libndctl
>> + * and papr_scm should provide backward compatibility until following two
>> + * assumptions/conditions when defining new PDSM structs hold:
>> + *
>> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
>> + *
>> + * 1. T(X) is a proper subset of T(Y) if Y > X.
>> + *    i.e Each new version of PDSM struct should retain existing struct
>> + *    attributes from previous version
>> + *
>> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
>> + *    it should also support T(1), T(2)...T(X - 1).
>> + *    i.e When adding support for new version of a PDSM struct, libndctl
>> + *    and papr_scm should retain support of the existing PDSM struct
>> + *    version they support.
>
> Please see this thread for an example why versions are a bad idea in UAPIs:
>
> https://lkml.org/lkml/2020/3/26/213
>

> While the use of version is different in that thread the fundamental issues are
> the same.  You end up with some weird matrix of supported features and
> structure definitions.  For example, you are opening up the possibility of
> changing structures with a different version for no good reason.

Not really sure I understand the statement correctly "you are opening up
the possibility of changing structures with a different version for no
good reason."
We want to return more data in the struct in future iterations. So
'changing structure with different version' is something we are
expecting. 

With the backward compatibility constraints 1 & 2 above, it will ensure
that support matrix looks like a lower traingular matrix with each
successive version supporting previous version attributes. So supporting
future versions is relatively simplified.

>
> Also having the user query with version Z and get back version X (older) is
> odd.  Generally if the kernel does not know about a feature (ie version Z of
> the structure) it should return -EINVAL and let the user figure out what to do.
> The user may just give up or they could try a different query.
>
Considering the flow of ndctl/libndctl this is needed. libndctl will
usually issues only one CMD_CALL ioctl to kernel and if that fails then
an error is reported and ndctl will exit loosing state.

Adding mechanism in libndctl to reissue CMD_CALL ioctl to fetch a
appropriate version of pdsm struct is going to be considerably more
work.

This version fall-back mechanism, ensures that libndctl will receive
usable data without having to reissue a more CMD_CALL ioctls.

>> + */
>> +
>> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
>> +struct nd_pdsm_cmd_pkg {
>> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
>> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
>> +	__u16 reserved[5];	/* Ignored and to be used in future */
>
> How do you know when reserved is used for something else in the future?  Is
> reserved guaranteed (and checked by the code) to be 0?

For current set of pdsm requests ignore these reserved fields. However a
future pdsm request can leverage these reserved fields. So papr_scm
just bind the usage of these fields with the value of
'nd_cmd_pkg.nd_command' that indicates the pdsm request.

That being said checking if the reserved fields are set to 0 will be a
good measure. Will add this check in next iteration.

>
>> +	__u16 payload_version;	/* In/Out: version of the payload */
>
> Why is payload_version after reserved?
Want to place the payload version field just before the payload data so
that it can be accessed with simple pointer arithmetic.

>
>> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
>> +} __packed;
>> +
>> +/*
>> + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
>> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
>> + */
>> +enum papr_pdsm {
>> +	PAPR_PDSM_MIN = 0x0,
>> +	PAPR_PDSM_MAX,
>> +};
>> +
>> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
>> +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
>> +{
>> +	return (struct nd_pdsm_cmd_pkg *) cmd;
>> +}
>> +
>> +/* Return the payload pointer for a given pcmd */
>> +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
>> +{
>> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
>> +		return NULL;
>> +	else
>> +		return (void *)(pcmd->payload);
>> +}
>> +
>> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
>> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
>> index 149431594839..5e2237e7ec08 100644
>> --- a/arch/powerpc/platforms/pseries/papr_scm.c
>> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
>> @@ -15,13 +15,15 @@
>>  #include <linux/seq_buf.h>
>>  
>>  #include <asm/plpar_wrappers.h>
>> +#include <asm/papr_pdsm.h>
>>  
>>  #define BIND_ANY_ADDR (~0ul)
>>  
>>  #define PAPR_SCM_DIMM_CMD_MASK \
>>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
>> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
>> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
>> +	 (1ul << ND_CMD_CALL))
>>  
>>  /* DIMM health bitmap bitmap indicators */
>>  /* SCM device is unable to persist memory contents */
>> @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>>  	return 0;
>>  }
>>  
>> +/*
>> + * Validate the inputs args to dimm-control function and return '0' if valid.
>> + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
>> + */
>> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>> +		       unsigned int buf_len)
>> +{
>> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
>> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
>> +	struct papr_scm_priv *p;
>> +
>> +	/* Only dimm-specific calls are supported atm */
>> +	if (!nvdimm)
>> +		return -EINVAL;
>> +
>> +	/* get the provider date from struct nvdimm */
>
> s/date/data
Thanks for point this out. Will fix this in next iteration.

>
>> +	p = nvdimm_provider_data(nvdimm);
>> +
>> +	if (!test_bit(cmd, &cmd_mask)) {
>> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
>> +		return -EINVAL;
>> +	} else if (cmd == ND_CMD_CALL) {
>> +
>> +		/* Verify the envelope package */
>> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
>> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
>> +				buf_len);
>> +			return -EINVAL;
>> +		}
>> +
>> +		/* Verify that the PDSM family is valid */
>> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
>> +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
>> +				pkg->hdr.nd_family);
>> +			return -EINVAL;
>> +
>> +		}
>> +
>> +		/* We except a payload with all PDSM commands */
>> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
>> +			dev_dbg(&p->pdev->dev,
>> +				"Empty payload for sub-command=0x%llx\n",
>> +				pkg->hdr.nd_command);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	/* Command looks valid */
>
> I assume the first command to be implemented also checks the { nd_command,
> payload_version, payload length } for correctness?
Yes the pdsm service functions do check the payload_version and
payload_length. Please see the papr_pdsm_health() that services the
PAPR_PDSM_HEALTH pdsm in Patch-5

>
>> +	return 0;
>> +}
>> +
>> +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
>> +				struct nd_pdsm_cmd_pkg *call_pkg)
>> +{
>> +	/* unknown subcommands return error in packages */
>> +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
>> +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
>> +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
>> +			call_pkg->hdr.nd_command);
>> +		call_pkg->cmd_status = -EINVAL;
>> +		return 0;
>> +	}
>> +
>> +	/* Depending on the DSM command call appropriate service routine */
>> +	switch (call_pkg->hdr.nd_command) {
>> +	default:
>> +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
>> +			call_pkg->hdr.nd_command);
>> +		call_pkg->cmd_status = -ENOENT;
>> +		return 0;
>> +	}
>> +}
>> +
>>  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>>  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>>  			  unsigned int buf_len, int *cmd_rc)
>>  {
>>  	struct nd_cmd_get_config_size *get_size_hdr;
>>  	struct papr_scm_priv *p;
>> +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
>> +	int rc;
>>  
>> -	/* Only dimm-specific calls are supported atm */
>> -	if (!nvdimm)
>> -		return -EINVAL;
>> +	/* Use a local variable in case cmd_rc pointer is NULL */
>> +	if (cmd_rc == NULL)
>> +		cmd_rc = &rc;
>
> Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
> and you did not change it.
This pointer is coming from outside the papr_scm code hence need to be
defensive here. Also as per[1] cmd_rc is "translation of firmware status"
and not every caller would need it hence making this pointer optional.

This is evident in acpi_nfit_blk_get_flags() where the 'nd_desc->ndctl'
is called with 'cmd_rc == NULL'.

[1] https://lore.kernel.org/linux-nvdimm/CAPcyv4hE_FG0YZXJVA1G=CBq8b9e0K54jxk5Sq5UKU-dnWT2Kg@mail.gmail.com/

>
>> +
>> +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
>> +	if (*cmd_rc) {
>> +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
>> +		return *cmd_rc;
>> +	}
>>  
>>  	p = nvdimm_provider_data(nvdimm);
>>  
>> @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>>  		*cmd_rc = papr_scm_meta_set(p, buf);
>>  		break;
>>  
>> +	case ND_CMD_CALL:
>> +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
>> +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
>> +		break;
>> +
>>  	default:
>> -		return -EINVAL;
>> +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
>> +		*cmd_rc = -EINVAL;
>
> Is this change related?  If there is a bug where there is a caller of
> papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
> that issue.
This simplifies a bit debugging of errors reported in
papr_scm_ndctl() as it ensures that subsequest dev_dbg "Returned with
cmd_rc" is always logged.

I think, this is a too small change to be carved out as an independent
patch. Also this doesnt change the behaviour of the code except logging
some more error info.

However, If you feel too strongly about it I will spin a separate patch
in this patch series for this.

>
> Ira
>
>>  	}
>>  
>>  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
>>  
>> -	return 0;
>> +	return *cmd_rc;
>>  }
>>  
>>  static ssize_t flags_show(struct device *dev,
>> diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
>> index de5d90212409..0e09dc5cec19 100644
>> --- a/include/uapi/linux/ndctl.h
>> +++ b/include/uapi/linux/ndctl.h
>> @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
>>  #define NVDIMM_FAMILY_HPE2 2
>>  #define NVDIMM_FAMILY_MSFT 3
>>  #define NVDIMM_FAMILY_HYPERV 4
>> +#define NVDIMM_FAMILY_PAPR 5
>>  
>>  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
>>  					struct nd_cmd_pkg)
>> -- 
>> 2.26.2
>>

Hi Ira,

Thanks again for reviewing this patch. My Response below:

Ira Weiny <ira.weiny@intel.com> writes:

> On Tue, Jun 02, 2020 at 01:51:49PM -0700, 'Ira Weiny' wrote:
>> On Tue, Jun 02, 2020 at 03:44:37PM +0530, Vaibhav Jain wrote:
>
> ...
>
>> > +
>> > +/*
>> > + * PDSM Envelope:
>> > + *
>> > + * The ioctl ND_CMD_CALL transfers data between user-space and kernel via
>> > + * envelope which consists of a header and user-defined payload sections.
>> > + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
>> > + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
>> > + * There is reserved field that can used to introduce new fields to the
>> > + * structure in future. It also tries to ensure that 'nd_pdsm_cmd_pkg.payload'
>> > + * lies at a 8-byte boundary.
>> > + *
>> > + *  +-------------+---------------------+---------------------------+
>> > + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
>> > + *  +-------------+---------------------+---------------------------+
>> > + *  |               nd_pdsm_cmd_pkg     |                           |
>> > + *  |-------------+                     |                           |
>> > + *  |  nd_cmd_pkg |                     |                           |
>> > + *  +-------------+---------------------+---------------------------+
>> > + *  | nd_family   |                     |                           |
>> > + *  | nd_size_out | cmd_status          |                           |
>> > + *  | nd_size_in  | payload_version     |     payload               |
>> > + *  | nd_command  | reserved            |                           |
>> > + *  | nd_fw_size  |                     |                           |
>> > + *  +-------------+---------------------+---------------------------+
>
> One more comment WRT nd_size_[in|out].  I know that it is defined as the size
> of the FW payload but normally when you nest headers 'size' in Header A
> represents everything after Header A, including Header B.  In this case that
> would be including nd_pdsm_cmd_pkg...
>
> It looks like that is not what you have done?  Or perhaps I missed it?
>
Not sure if I understand the question correctly.
'struct nd_pdsm_cmd_pkg' contains 'struct nd_cmd_pkg' at its head and
its size_[in|out] are populated by the libndctl in userspace, setting
them to data following the 'struct nd_cmd_pkg'.

Copying of 'struct nd_cmd_pkg' to the input/out envelop is implicitly
done in __nd_ioctl via the command descriptor array
__nd_cmd_bus_descs. So I dont need to add the size of 'struct
nd_cmd_pkg' to nd_size_[in|out].

> Ira
>
>> > + *
>> > + * PDSM Header:
>> > + *
>> > + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
>> > + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to member
>> > + * 'nd_cmd_pkg.nd_command'. Apart from size information of the envelope which is
>> > + * contained in 'struct nd_cmd_pkg', the header also has members following 
>>                                                              ^^^^^
>>                                                         ...  the  ...
>> 
>> > + * members:
>> > + *
>> > + * 'cmd_status'		: (Out) Errors if any encountered while servicing PDSM.
>> > + * 'payload_version'	: (In/Out) Version number associated with the payload.
>> > + * 'reserved'		: Not used and reserved for future.
>> > + *
>> > + * PDSM Payload:
>> > + *
>> > + * The layout of the PDSM Payload is defined by various structs shared between
>> > + * papr_scm and libndctl so that contents of payload can be interpreted. During
>> > + * servicing of a PDSM the papr_scm module will read input args from the payload
>> > + * field by casting its contents to an appropriate struct pointer based on the
>> > + * PDSM command. Similarly the output of servicing the PDSM command will be
>> > + * copied to the payload field using the same struct.
>> > + *
>> > + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size, which
>> > + * leaves around 176 bytes for the envelope payload (ignoring any padding that
>> > + * the compiler may silently introduce).
>> > + *
>> > + * Payload Version:
>> > + *
>> > + * A 'payload_version' field is present in PDSM header that indicates a specific
>> > + * version of the structure present in PDSM Payload for a given PDSM command.
>> > + * This provides backward compatibility in case the PDSM Payload structure
>> > + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
>> > + *
>> > + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
>> > + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
>> > + * module when servicing the PDSM envelope checks the 'payload_version' and then
>> > + * uses 'payload struct version' == MIN('payload_version field',
>> > + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
>> > + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
>> > + * struct in returned 'payload_version' field.
>> > + *
>> > + * Libndctl on receiving the envelope back from papr_scm again checks the
>> > + * 'payload_version' field and based on it use the appropriate version dsm
>> > + * struct to parse the results.
>> > + *
>> > + * Backward Compatibility:
>> > + *
>> > + * Above scheme of exchanging different versioned PDSM struct between libndctl
>> > + * and papr_scm should provide backward compatibility until following two
>> > + * assumptions/conditions when defining new PDSM structs hold:
>> > + *
>> > + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
>> > + *
>> > + * 1. T(X) is a proper subset of T(Y) if Y > X.
>> > + *    i.e Each new version of PDSM struct should retain existing struct
>> > + *    attributes from previous version
>> > + *
>> > + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
>> > + *    it should also support T(1), T(2)...T(X - 1).
>> > + *    i.e When adding support for new version of a PDSM struct, libndctl
>> > + *    and papr_scm should retain support of the existing PDSM struct
>> > + *    version they support.
>> 
>> Please see this thread for an example why versions are a bad idea in UAPIs:
>> 
>> https://lkml.org/lkml/2020/3/26/213
>> 
>> While the use of version is different in that thread the fundamental issues are
>> the same.  You end up with some weird matrix of supported features and
>> structure definitions.  For example, you are opening up the possibility of
>> changing structures with a different version for no good reason.
>> 
>> Also having the user query with version Z and get back version X (older) is
>> odd.  Generally if the kernel does not know about a feature (ie version Z of
>> the structure) it should return -EINVAL and let the user figure out what to do.
>> The user may just give up or they could try a different query.
>> 
>> > + */
>> > +
>> > +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
>> > +struct nd_pdsm_cmd_pkg {
>> > +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
>> > +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
>> > +	__u16 reserved[5];	/* Ignored and to be used in future */
>> 
>> How do you know when reserved is used for something else in the future?  Is
>> reserved guaranteed (and checked by the code) to be 0?
>> 
>> > +	__u16 payload_version;	/* In/Out: version of the payload */
>> 
>> Why is payload_version after reserved?
>> 
>> > +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
>> > +} __packed;
>> > +
>> > +/*
>> > + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
>> > + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
>> > + */
>> > +enum papr_pdsm {
>> > +	PAPR_PDSM_MIN = 0x0,
>> > +	PAPR_PDSM_MAX,
>> > +};
>> > +
>> > +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
>> > +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
>> > +{
>> > +	return (struct nd_pdsm_cmd_pkg *) cmd;
>> > +}
>> > +
>> > +/* Return the payload pointer for a given pcmd */
>> > +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
>> > +{
>> > +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
>> > +		return NULL;
>> > +	else
>> > +		return (void *)(pcmd->payload);
>> > +}
>> > +
>> > +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
>> > diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
>> > index 149431594839..5e2237e7ec08 100644
>> > --- a/arch/powerpc/platforms/pseries/papr_scm.c
>> > +++ b/arch/powerpc/platforms/pseries/papr_scm.c
>> > @@ -15,13 +15,15 @@
>> >  #include <linux/seq_buf.h>
>> >  
>> >  #include <asm/plpar_wrappers.h>
>> > +#include <asm/papr_pdsm.h>
>> >  
>> >  #define BIND_ANY_ADDR (~0ul)
>> >  
>> >  #define PAPR_SCM_DIMM_CMD_MASK \
>> >  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>> >  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
>> > -	 (1ul << ND_CMD_SET_CONFIG_DATA))
>> > +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
>> > +	 (1ul << ND_CMD_CALL))
>> >  
>> >  /* DIMM health bitmap bitmap indicators */
>> >  /* SCM device is unable to persist memory contents */
>> > @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>> >  	return 0;
>> >  }
>> >  
>> > +/*
>> > + * Validate the inputs args to dimm-control function and return '0' if valid.
>> > + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
>> > + */
>> > +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>> > +		       unsigned int buf_len)
>> > +{
>> > +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
>> > +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
>> > +	struct papr_scm_priv *p;
>> > +
>> > +	/* Only dimm-specific calls are supported atm */
>> > +	if (!nvdimm)
>> > +		return -EINVAL;
>> > +
>> > +	/* get the provider date from struct nvdimm */
>> 
>> s/date/data
>> 
>> > +	p = nvdimm_provider_data(nvdimm);
>> > +
>> > +	if (!test_bit(cmd, &cmd_mask)) {
>> > +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
>> > +		return -EINVAL;
>> > +	} else if (cmd == ND_CMD_CALL) {
>> > +
>> > +		/* Verify the envelope package */
>> > +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
>> > +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
>> > +				buf_len);
>> > +			return -EINVAL;
>> > +		}
>> > +
>> > +		/* Verify that the PDSM family is valid */
>> > +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
>> > +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
>> > +				pkg->hdr.nd_family);
>> > +			return -EINVAL;
>> > +
>> > +		}
>> > +
>> > +		/* We except a payload with all PDSM commands */
>> > +		if (pdsm_cmd_to_payload(pkg) == NULL) {
>> > +			dev_dbg(&p->pdev->dev,
>> > +				"Empty payload for sub-command=0x%llx\n",
>> > +				pkg->hdr.nd_command);
>> > +			return -EINVAL;
>> > +		}
>> > +	}
>> > +
>> > +	/* Command looks valid */
>> 
>> I assume the first command to be implemented also checks the { nd_command,
>> payload_version, payload length } for correctness?
>> 
>> > +	return 0;
>> > +}
>> > +
>> > +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
>> > +				struct nd_pdsm_cmd_pkg *call_pkg)
>> > +{
>> > +	/* unknown subcommands return error in packages */
>> > +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
>> > +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
>> > +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
>> > +			call_pkg->hdr.nd_command);
>> > +		call_pkg->cmd_status = -EINVAL;
>> > +		return 0;
>> > +	}
>> > +
>> > +	/* Depending on the DSM command call appropriate service routine */
>> > +	switch (call_pkg->hdr.nd_command) {
>> > +	default:
>> > +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
>> > +			call_pkg->hdr.nd_command);
>> > +		call_pkg->cmd_status = -ENOENT;
>> > +		return 0;
>> > +	}
>> > +}
>> > +
>> >  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>> >  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>> >  			  unsigned int buf_len, int *cmd_rc)
>> >  {
>> >  	struct nd_cmd_get_config_size *get_size_hdr;
>> >  	struct papr_scm_priv *p;
>> > +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
>> > +	int rc;
>> >  
>> > -	/* Only dimm-specific calls are supported atm */
>> > -	if (!nvdimm)
>> > -		return -EINVAL;
>> > +	/* Use a local variable in case cmd_rc pointer is NULL */
>> > +	if (cmd_rc == NULL)
>> > +		cmd_rc = &rc;
>> 
>> Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
>> and you did not change it.
>> 
>> > +
>> > +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
>> > +	if (*cmd_rc) {
>> > +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
>> > +		return *cmd_rc;
>> > +	}
>> >  
>> >  	p = nvdimm_provider_data(nvdimm);
>> >  
>> > @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>> >  		*cmd_rc = papr_scm_meta_set(p, buf);
>> >  		break;
>> >  
>> > +	case ND_CMD_CALL:
>> > +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
>> > +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
>> > +		break;
>> > +
>> >  	default:
>> > -		return -EINVAL;
>> > +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
>> > +		*cmd_rc = -EINVAL;
>> 
>> Is this change related?  If there is a bug where there is a caller of
>> papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
>> that issue.
>> 
>> Ira
>> 
>> >  	}
>> >  
>> >  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
>> >  
>> > -	return 0;
>> > +	return *cmd_rc;
>> >  }
>> >  
>> >  static ssize_t flags_show(struct device *dev,
>> > diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
>> > index de5d90212409..0e09dc5cec19 100644
>> > --- a/include/uapi/linux/ndctl.h
>> > +++ b/include/uapi/linux/ndctl.h
>> > @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
>> >  #define NVDIMM_FAMILY_HPE2 2
>> >  #define NVDIMM_FAMILY_MSFT 3
>> >  #define NVDIMM_FAMILY_HYPERV 4
>> > +#define NVDIMM_FAMILY_PAPR 5
>> >  
>> >  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
>> >  					struct nd_cmd_pkg)
>> > -- 
>> > 2.26.2
>> > 
>> _______________________________________________
>> Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
>> To unsubscribe send an email to linux-nvdimm-leave@lists.01.org

On Wed, Jun 03, 2020 at 11:41:42PM +0530, Vaibhav Jain wrote:
> Hi Ira,
> 
> Thanks for reviewing this patch. My responses below:
> 
> Ira Weiny <ira.weiny@intel.com> writes:
> 

...

> >> + *
> >> + * Payload Version:
> >> + *
> >> + * A 'payload_version' field is present in PDSM header that indicates a specific
> >> + * version of the structure present in PDSM Payload for a given PDSM command.
> >> + * This provides backward compatibility in case the PDSM Payload structure
> >> + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
> >> + *
> >> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
> >> + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
> >> + * module when servicing the PDSM envelope checks the 'payload_version' and then
> >> + * uses 'payload struct version' == MIN('payload_version field',
> >> + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
> >> + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
> >> + * struct in returned 'payload_version' field.
> >> + *
> >> + * Libndctl on receiving the envelope back from papr_scm again checks the
> >> + * 'payload_version' field and based on it use the appropriate version dsm
> >> + * struct to parse the results.
> >> + *
> >> + * Backward Compatibility:
> >> + *
> >> + * Above scheme of exchanging different versioned PDSM struct between libndctl
> >> + * and papr_scm should provide backward compatibility until following two
> >> + * assumptions/conditions when defining new PDSM structs hold:
> >> + *
> >> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
> >> + *
> >> + * 1. T(X) is a proper subset of T(Y) if Y > X.
> >> + *    i.e Each new version of PDSM struct should retain existing struct
> >> + *    attributes from previous version
> >> + *
> >> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
> >> + *    it should also support T(1), T(2)...T(X - 1).
> >> + *    i.e When adding support for new version of a PDSM struct, libndctl
> >> + *    and papr_scm should retain support of the existing PDSM struct
> >> + *    version they support.
> >
> > Please see this thread for an example why versions are a bad idea in UAPIs:
> >
> > https://lkml.org/lkml/2020/3/26/213
> >
> 
> > While the use of version is different in that thread the fundamental issues are
> > the same.  You end up with some weird matrix of supported features and
> > structure definitions.  For example, you are opening up the possibility of
> > changing structures with a different version for no good reason.
> 
> Not really sure I understand the statement correctly "you are opening up
> the possibility of changing structures with a different version for no
> good reason."

What I mean is:

struct v1 {
	u32 x;
	u32 y;
};

struct v2 {
	u32 y;
	u32 x;
};

x and y are the same data but you have now redefined the order of the struct.
You don't need that flexibility/complexity.

Generally I think you are defining:

struct v1 {
	u32 x;
	u32 y;
};

struct v2 {
	u32 x;
	u32 y;
	u32 z;
	u32 a;
};

Which becomes 2 structures...  There is no need.

The easiest thing to do is:

struct user_data {
	u32 x;
	u32 y;
};

And later you modify user_data to:

struct user_data {
	u32 x;
	u32 y;
	u32 z;
	u32 a;
};

libndctl always passes sizeof(struct user_data) to the call. [Do ensure
structures are 64bit aligned for this to work.]

The kernel sees the size and returns the amount of data up to that size.

Therefore, older kernels automatically fill in x and y,  newer kernels fill in
z/a if the buffer was big enough.  libndctl only uses the fields it knows about.

It is _much_ easier this way.  Almost nothing needs to get changed as versions
roll forward.  The only big issue is if libndctl _needs_ z then it has to check
if z is returned.

In that case add a cap_mask with bit fields which the kernel can fill in for
which fields are valid.

struct user_data {
	u64 cap_mask;  /* where bits define extra future capabilities */
	u32 x;
	u32 y;
};

IFF you need to add data within fields which are reserved you can use
capability flags to indicate which fields are requested and which are returned
by the kernel.

But I _think_ for what you want libndctl must survive if z/a are not available
right?  So just adding to the structure should be fine.

> We want to return more data in the struct in future iterations. So
> 'changing structure with different version' is something we are
> expecting. 
> 
> With the backward compatibility constraints 1 & 2 above, it will ensure
> that support matrix looks like a lower traingular matrix with each
> successive version supporting previous version attributes. So supporting
> future versions is relatively simplified.

But you end up with weird switch/if's etc to deal with the multiple structures.

With the size method the kernel simply returns the same size data as the user
requested and everything is done.  No logic required at all.  Literally it can
just copy the data it has (truncating if necessary).

> 
> >
> > Also having the user query with version Z and get back version X (older) is
> > odd.  Generally if the kernel does not know about a feature (ie version Z of
> > the structure) it should return -EINVAL and let the user figure out what to do.
> > The user may just give up or they could try a different query.
> >
> Considering the flow of ndctl/libndctl this is needed. libndctl will
> usually issues only one CMD_CALL ioctl to kernel and if that fails then
> an error is reported and ndctl will exit loosing state.
> 
> Adding mechanism in libndctl to reissue CMD_CALL ioctl to fetch a
> appropriate version of pdsm struct is going to be considerably more
> work.
> 
> This version fall-back mechanism, ensures that libndctl will receive
> usable data without having to reissue a more CMD_CALL ioctls.

Define usable?

What happens if libndctl does not get 'z' in my example above?  What does it
do?  If I understand correctly it does not _need_ z.  So why have a check on
the version from the kernel?

What if we change to:

struct v3 {
	u32 x;
	u32 y;
	u32 z;
	u32 a;
	u32 b;
	u32 c;
};

Now it has to

	if(version 2)
		z/a valid do something()

	if(version 3)
		b/c valid do something else()

if z, a, b, c are all 0 does it matter?

If not, the logic above disappears.

If so, then you need a cap mask.  Then the kernel can say c and a are valid
(but c is 0) or other flexible stuff like that.

> 
> >> + */
> >> +
> >> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
> >> +struct nd_pdsm_cmd_pkg {
> >> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
> >> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
> >> +	__u16 reserved[5];	/* Ignored and to be used in future */
> >
> > How do you know when reserved is used for something else in the future?  Is
> > reserved guaranteed (and checked by the code) to be 0?
> 
> For current set of pdsm requests ignore these reserved fields. However a
> future pdsm request can leverage these reserved fields. So papr_scm
> just bind the usage of these fields with the value of
> 'nd_cmd_pkg.nd_command' that indicates the pdsm request.
> 
> That being said checking if the reserved fields are set to 0 will be a
> good measure. Will add this check in next iteration.

Exactly, if you don't check them now you will end up with an older libndctl
which passes in garbage and breaks future users...  Basically rendering the
reserved fields useless.

> 
> >
> >> +	__u16 payload_version;	/* In/Out: version of the payload */
> >
> > Why is payload_version after reserved?
> Want to place the payload version field just before the payload data so
> that it can be accessed with simple pointer arithmetic.

I did not see that in the patch.  I thought you were using
nd_pdsm_cmd_pkg->payload_version?

> 
> >
> >> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
> >> +} __packed;
> >> +
> >> +/*
> >> + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
> >> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
> >> + */
> >> +enum papr_pdsm {
> >> +	PAPR_PDSM_MIN = 0x0,
> >> +	PAPR_PDSM_MAX,
> >> +};
> >> +
> >> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
> >> +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
> >> +{
> >> +	return (struct nd_pdsm_cmd_pkg *) cmd;
> >> +}
> >> +
> >> +/* Return the payload pointer for a given pcmd */
> >> +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
> >> +{
> >> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
> >> +		return NULL;
> >> +	else
> >> +		return (void *)(pcmd->payload);
> >> +}
> >> +
> >> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
> >> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
> >> index 149431594839..5e2237e7ec08 100644
> >> --- a/arch/powerpc/platforms/pseries/papr_scm.c
> >> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> >> @@ -15,13 +15,15 @@
> >>  #include <linux/seq_buf.h>
> >>  
> >>  #include <asm/plpar_wrappers.h>
> >> +#include <asm/papr_pdsm.h>
> >>  
> >>  #define BIND_ANY_ADDR (~0ul)
> >>  
> >>  #define PAPR_SCM_DIMM_CMD_MASK \
> >>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
> >>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
> >> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
> >> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
> >> +	 (1ul << ND_CMD_CALL))
> >>  
> >>  /* DIMM health bitmap bitmap indicators */
> >>  /* SCM device is unable to persist memory contents */
> >> @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
> >>  	return 0;
> >>  }
> >>  
> >> +/*
> >> + * Validate the inputs args to dimm-control function and return '0' if valid.
> >> + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
> >> + */
> >> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> >> +		       unsigned int buf_len)
> >> +{
> >> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
> >> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
> >> +	struct papr_scm_priv *p;
> >> +
> >> +	/* Only dimm-specific calls are supported atm */
> >> +	if (!nvdimm)
> >> +		return -EINVAL;
> >> +
> >> +	/* get the provider date from struct nvdimm */
> >
> > s/date/data
> Thanks for point this out. Will fix this in next iteration.
> 
> >
> >> +	p = nvdimm_provider_data(nvdimm);
> >> +
> >> +	if (!test_bit(cmd, &cmd_mask)) {
> >> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
> >> +		return -EINVAL;
> >> +	} else if (cmd == ND_CMD_CALL) {
> >> +
> >> +		/* Verify the envelope package */
> >> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
> >> +				buf_len);
> >> +			return -EINVAL;
> >> +		}
> >> +
> >> +		/* Verify that the PDSM family is valid */
> >> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
> >> +				pkg->hdr.nd_family);
> >> +			return -EINVAL;
> >> +
> >> +		}
> >> +
> >> +		/* We except a payload with all PDSM commands */
> >> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
> >> +			dev_dbg(&p->pdev->dev,
> >> +				"Empty payload for sub-command=0x%llx\n",
> >> +				pkg->hdr.nd_command);
> >> +			return -EINVAL;
> >> +		}
> >> +	}
> >> +
> >> +	/* Command looks valid */
> >
> > I assume the first command to be implemented also checks the { nd_command,
> > payload_version, payload length } for correctness?
> Yes the pdsm service functions do check the payload_version and
> payload_length. Please see the papr_pdsm_health() that services the
> PAPR_PDSM_HEALTH pdsm in Patch-5
> 

cool.

> >
> >> +	return 0;
> >> +}
> >> +
> >> +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
> >> +				struct nd_pdsm_cmd_pkg *call_pkg)
> >> +{
> >> +	/* unknown subcommands return error in packages */
> >> +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
> >> +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
> >> +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
> >> +			call_pkg->hdr.nd_command);
> >> +		call_pkg->cmd_status = -EINVAL;
> >> +		return 0;
> >> +	}
> >> +
> >> +	/* Depending on the DSM command call appropriate service routine */
> >> +	switch (call_pkg->hdr.nd_command) {
> >> +	default:
> >> +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
> >> +			call_pkg->hdr.nd_command);
> >> +		call_pkg->cmd_status = -ENOENT;
> >> +		return 0;
> >> +	}
> >> +}
> >> +
> >>  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
> >>  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> >>  			  unsigned int buf_len, int *cmd_rc)
> >>  {
> >>  	struct nd_cmd_get_config_size *get_size_hdr;
> >>  	struct papr_scm_priv *p;
> >> +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
> >> +	int rc;
> >>  
> >> -	/* Only dimm-specific calls are supported atm */
> >> -	if (!nvdimm)
> >> -		return -EINVAL;
> >> +	/* Use a local variable in case cmd_rc pointer is NULL */
> >> +	if (cmd_rc == NULL)
> >> +		cmd_rc = &rc;
> >
> > Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
> > and you did not change it.
> This pointer is coming from outside the papr_scm code hence need to be
> defensive here. Also as per[1] cmd_rc is "translation of firmware status"
> and not every caller would need it hence making this pointer optional.
> 
> This is evident in acpi_nfit_blk_get_flags() where the 'nd_desc->ndctl'
> is called with 'cmd_rc == NULL'.
> 
> [1] https://lore.kernel.org/linux-nvdimm/CAPcyv4hE_FG0YZXJVA1G=CBq8b9e0K54jxk5Sq5UKU-dnWT2Kg@mail.gmail.com/

Ah... Ok.  So this is a bug fix which needs to happen regardless of the status
of this patch...

> 
> >
> >> +
> >> +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
> >> +	if (*cmd_rc) {
> >> +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
> >> +		return *cmd_rc;
> >> +	}
> >>  
> >>  	p = nvdimm_provider_data(nvdimm);
> >>  
> >> @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
> >>  		*cmd_rc = papr_scm_meta_set(p, buf);

... Because this will break here. even without this new code...  right?

Lets get this fix in as a prelim-patch.

> >>  		break;
> >>  
> >> +	case ND_CMD_CALL:
> >> +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
> >> +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
> >> +		break;
> >> +
> >>  	default:
> >> -		return -EINVAL;
> >> +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
> >> +		*cmd_rc = -EINVAL;
> >
> > Is this change related?  If there is a bug where there is a caller of
> > papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
> > that issue.
> This simplifies a bit debugging of errors reported in
> papr_scm_ndctl() as it ensures that subsequest dev_dbg "Returned with
> cmd_rc" is always logged.
> 
> I think, this is a too small change to be carved out as an independent
> patch. Also this doesnt change the behaviour of the code except logging
> some more error info.
> 
> However, If you feel too strongly about it I will spin a separate patch
> in this patch series for this.

This can go in as part of a 'protect against cmd_rc == NULL' preliminary patch.

I flagged this because at first I could not figure out what this had to do with
the ND_CMD_CALL...

For reviewers you want to make your patches concise to what you are
fixing/adding...

Also, based on acpi_nfit_blk_get_flags() using cmd_rc == NULL it looks like we
have a bug which needs to get fixed regardless of the this patch.  And if that
bug exists in earlier kernels you will need a separate patch to backport as a
fix.

So lets get that in first and separate...  :-D

Ira

> 
> >
> > Ira
> >
> >>  	}
> >>  
> >>  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
> >>  
> >> -	return 0;
> >> +	return *cmd_rc;
> >>  }
> >>  
> >>  static ssize_t flags_show(struct device *dev,
> >> diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
> >> index de5d90212409..0e09dc5cec19 100644
> >> --- a/include/uapi/linux/ndctl.h
> >> +++ b/include/uapi/linux/ndctl.h
> >> @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
> >>  #define NVDIMM_FAMILY_HPE2 2
> >>  #define NVDIMM_FAMILY_MSFT 3
> >>  #define NVDIMM_FAMILY_HYPERV 4
> >> +#define NVDIMM_FAMILY_PAPR 5
> >>  
> >>  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
> >>  					struct nd_cmd_pkg)
> >> -- 
> >> 2.26.2
> >> 
> 
> -- 
> Cheers
> ~ Vaibhav

On Thu, Jun 04, 2020 at 01:58:00AM +0530, Vaibhav Jain wrote:
> Hi Ira,
> 
> Thanks again for reviewing this patch. My Response below:
> 
> Ira Weiny <ira.weiny@intel.com> writes:
> 
> > On Tue, Jun 02, 2020 at 01:51:49PM -0700, 'Ira Weiny' wrote:
> >> On Tue, Jun 02, 2020 at 03:44:37PM +0530, Vaibhav Jain wrote:
> >
> > ...
> >
> >> > +
> >> > +/*
> >> > + * PDSM Envelope:
> >> > + *
> >> > + * The ioctl ND_CMD_CALL transfers data between user-space and kernel via
> >> > + * envelope which consists of a header and user-defined payload sections.
> >> > + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
> >> > + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
> >> > + * There is reserved field that can used to introduce new fields to the
> >> > + * structure in future. It also tries to ensure that 'nd_pdsm_cmd_pkg.payload'
> >> > + * lies at a 8-byte boundary.
> >> > + *
> >> > + *  +-------------+---------------------+---------------------------+
> >> > + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
> >> > + *  +-------------+---------------------+---------------------------+
> >> > + *  |               nd_pdsm_cmd_pkg     |                           |
> >> > + *  |-------------+                     |                           |
> >> > + *  |  nd_cmd_pkg |                     |                           |
> >> > + *  +-------------+---------------------+---------------------------+
> >> > + *  | nd_family   |                     |                           |
> >> > + *  | nd_size_out | cmd_status          |                           |
> >> > + *  | nd_size_in  | payload_version     |     payload               |
> >> > + *  | nd_command  | reserved            |                           |
> >> > + *  | nd_fw_size  |                     |                           |
> >> > + *  +-------------+---------------------+---------------------------+
> >
> > One more comment WRT nd_size_[in|out].  I know that it is defined as the size
> > of the FW payload but normally when you nest headers 'size' in Header A
> > represents everything after Header A, including Header B.  In this case that
> > would be including nd_pdsm_cmd_pkg...
> >
> > It looks like that is not what you have done?  Or perhaps I missed it?
> >
> Not sure if I understand the question correctly.
> 'struct nd_pdsm_cmd_pkg' contains 'struct nd_cmd_pkg' at its head and
> its size_[in|out] are populated by the libndctl in userspace, setting
> them to data following the 'struct nd_cmd_pkg'.
> 
> Copying of 'struct nd_cmd_pkg' to the input/out envelop is implicitly
> done in __nd_ioctl via the command descriptor array
> __nd_cmd_bus_descs. So I dont need to add the size of 'struct
> nd_cmd_pkg' to nd_size_[in|out].

Yea I see that now...  Coming from a networking background I find that odd...
:-/  Usually 'size' in a header includes all data after that header.  Because
header A knows nothing of the rest of the 'payload'...

FWIW you could define nd_size_in anyway you want because you are not really
sending any payload back from firmware directly.  But I suppose I can live with
it.

Ira

> 
> > Ira
> >
> >> > + *
> >> > + * PDSM Header:
> >> > + *
> >> > + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
> >> > + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to member
> >> > + * 'nd_cmd_pkg.nd_command'. Apart from size information of the envelope which is
> >> > + * contained in 'struct nd_cmd_pkg', the header also has members following 
> >>                                                              ^^^^^
> >>                                                         ...  the  ...
> >> 
> >> > + * members:
> >> > + *
> >> > + * 'cmd_status'		: (Out) Errors if any encountered while servicing PDSM.
> >> > + * 'payload_version'	: (In/Out) Version number associated with the payload.
> >> > + * 'reserved'		: Not used and reserved for future.
> >> > + *
> >> > + * PDSM Payload:
> >> > + *
> >> > + * The layout of the PDSM Payload is defined by various structs shared between
> >> > + * papr_scm and libndctl so that contents of payload can be interpreted. During
> >> > + * servicing of a PDSM the papr_scm module will read input args from the payload
> >> > + * field by casting its contents to an appropriate struct pointer based on the
> >> > + * PDSM command. Similarly the output of servicing the PDSM command will be
> >> > + * copied to the payload field using the same struct.
> >> > + *
> >> > + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size, which
> >> > + * leaves around 176 bytes for the envelope payload (ignoring any padding that
> >> > + * the compiler may silently introduce).
> >> > + *
> >> > + * Payload Version:
> >> > + *
> >> > + * A 'payload_version' field is present in PDSM header that indicates a specific
> >> > + * version of the structure present in PDSM Payload for a given PDSM command.
> >> > + * This provides backward compatibility in case the PDSM Payload structure
> >> > + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
> >> > + *
> >> > + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
> >> > + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
> >> > + * module when servicing the PDSM envelope checks the 'payload_version' and then
> >> > + * uses 'payload struct version' == MIN('payload_version field',
> >> > + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
> >> > + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
> >> > + * struct in returned 'payload_version' field.
> >> > + *
> >> > + * Libndctl on receiving the envelope back from papr_scm again checks the
> >> > + * 'payload_version' field and based on it use the appropriate version dsm
> >> > + * struct to parse the results.
> >> > + *
> >> > + * Backward Compatibility:
> >> > + *
> >> > + * Above scheme of exchanging different versioned PDSM struct between libndctl
> >> > + * and papr_scm should provide backward compatibility until following two
> >> > + * assumptions/conditions when defining new PDSM structs hold:
> >> > + *
> >> > + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
> >> > + *
> >> > + * 1. T(X) is a proper subset of T(Y) if Y > X.
> >> > + *    i.e Each new version of PDSM struct should retain existing struct
> >> > + *    attributes from previous version
> >> > + *
> >> > + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
> >> > + *    it should also support T(1), T(2)...T(X - 1).
> >> > + *    i.e When adding support for new version of a PDSM struct, libndctl
> >> > + *    and papr_scm should retain support of the existing PDSM struct
> >> > + *    version they support.
> >> 
> >> Please see this thread for an example why versions are a bad idea in UAPIs:
> >> 
> >> https://lkml.org/lkml/2020/3/26/213
> >> 
> >> While the use of version is different in that thread the fundamental issues are
> >> the same.  You end up with some weird matrix of supported features and
> >> structure definitions.  For example, you are opening up the possibility of
> >> changing structures with a different version for no good reason.
> >> 
> >> Also having the user query with version Z and get back version X (older) is
> >> odd.  Generally if the kernel does not know about a feature (ie version Z of
> >> the structure) it should return -EINVAL and let the user figure out what to do.
> >> The user may just give up or they could try a different query.
> >> 
> >> > + */
> >> > +
> >> > +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
> >> > +struct nd_pdsm_cmd_pkg {
> >> > +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
> >> > +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
> >> > +	__u16 reserved[5];	/* Ignored and to be used in future */
> >> 
> >> How do you know when reserved is used for something else in the future?  Is
> >> reserved guaranteed (and checked by the code) to be 0?
> >> 
> >> > +	__u16 payload_version;	/* In/Out: version of the payload */
> >> 
> >> Why is payload_version after reserved?
> >> 
> >> > +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
> >> > +} __packed;
> >> > +
> >> > +/*
> >> > + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
> >> > + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
> >> > + */
> >> > +enum papr_pdsm {
> >> > +	PAPR_PDSM_MIN = 0x0,
> >> > +	PAPR_PDSM_MAX,
> >> > +};
> >> > +
> >> > +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
> >> > +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
> >> > +{
> >> > +	return (struct nd_pdsm_cmd_pkg *) cmd;
> >> > +}
> >> > +
> >> > +/* Return the payload pointer for a given pcmd */
> >> > +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
> >> > +{
> >> > +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
> >> > +		return NULL;
> >> > +	else
> >> > +		return (void *)(pcmd->payload);
> >> > +}
> >> > +
> >> > +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
> >> > diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
> >> > index 149431594839..5e2237e7ec08 100644
> >> > --- a/arch/powerpc/platforms/pseries/papr_scm.c
> >> > +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> >> > @@ -15,13 +15,15 @@
> >> >  #include <linux/seq_buf.h>
> >> >  
> >> >  #include <asm/plpar_wrappers.h>
> >> > +#include <asm/papr_pdsm.h>
> >> >  
> >> >  #define BIND_ANY_ADDR (~0ul)
> >> >  
> >> >  #define PAPR_SCM_DIMM_CMD_MASK \
> >> >  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
> >> >  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
> >> > -	 (1ul << ND_CMD_SET_CONFIG_DATA))
> >> > +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
> >> > +	 (1ul << ND_CMD_CALL))
> >> >  
> >> >  /* DIMM health bitmap bitmap indicators */
> >> >  /* SCM device is unable to persist memory contents */
> >> > @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
> >> >  	return 0;
> >> >  }
> >> >  
> >> > +/*
> >> > + * Validate the inputs args to dimm-control function and return '0' if valid.
> >> > + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
> >> > + */
> >> > +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> >> > +		       unsigned int buf_len)
> >> > +{
> >> > +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
> >> > +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
> >> > +	struct papr_scm_priv *p;
> >> > +
> >> > +	/* Only dimm-specific calls are supported atm */
> >> > +	if (!nvdimm)
> >> > +		return -EINVAL;
> >> > +
> >> > +	/* get the provider date from struct nvdimm */
> >> 
> >> s/date/data
> >> 
> >> > +	p = nvdimm_provider_data(nvdimm);
> >> > +
> >> > +	if (!test_bit(cmd, &cmd_mask)) {
> >> > +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
> >> > +		return -EINVAL;
> >> > +	} else if (cmd == ND_CMD_CALL) {
> >> > +
> >> > +		/* Verify the envelope package */
> >> > +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
> >> > +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
> >> > +				buf_len);
> >> > +			return -EINVAL;
> >> > +		}
> >> > +
> >> > +		/* Verify that the PDSM family is valid */
> >> > +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
> >> > +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
> >> > +				pkg->hdr.nd_family);
> >> > +			return -EINVAL;
> >> > +
> >> > +		}
> >> > +
> >> > +		/* We except a payload with all PDSM commands */
> >> > +		if (pdsm_cmd_to_payload(pkg) == NULL) {
> >> > +			dev_dbg(&p->pdev->dev,
> >> > +				"Empty payload for sub-command=0x%llx\n",
> >> > +				pkg->hdr.nd_command);
> >> > +			return -EINVAL;
> >> > +		}
> >> > +	}
> >> > +
> >> > +	/* Command looks valid */
> >> 
> >> I assume the first command to be implemented also checks the { nd_command,
> >> payload_version, payload length } for correctness?
> >> 
> >> > +	return 0;
> >> > +}
> >> > +
> >> > +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
> >> > +				struct nd_pdsm_cmd_pkg *call_pkg)
> >> > +{
> >> > +	/* unknown subcommands return error in packages */
> >> > +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
> >> > +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
> >> > +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
> >> > +			call_pkg->hdr.nd_command);
> >> > +		call_pkg->cmd_status = -EINVAL;
> >> > +		return 0;
> >> > +	}
> >> > +
> >> > +	/* Depending on the DSM command call appropriate service routine */
> >> > +	switch (call_pkg->hdr.nd_command) {
> >> > +	default:
> >> > +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
> >> > +			call_pkg->hdr.nd_command);
> >> > +		call_pkg->cmd_status = -ENOENT;
> >> > +		return 0;
> >> > +	}
> >> > +}
> >> > +
> >> >  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
> >> >  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
> >> >  			  unsigned int buf_len, int *cmd_rc)
> >> >  {
> >> >  	struct nd_cmd_get_config_size *get_size_hdr;
> >> >  	struct papr_scm_priv *p;
> >> > +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
> >> > +	int rc;
> >> >  
> >> > -	/* Only dimm-specific calls are supported atm */
> >> > -	if (!nvdimm)
> >> > -		return -EINVAL;
> >> > +	/* Use a local variable in case cmd_rc pointer is NULL */
> >> > +	if (cmd_rc == NULL)
> >> > +		cmd_rc = &rc;
> >> 
> >> Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
> >> and you did not change it.
> >> 
> >> > +
> >> > +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
> >> > +	if (*cmd_rc) {
> >> > +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
> >> > +		return *cmd_rc;
> >> > +	}
> >> >  
> >> >  	p = nvdimm_provider_data(nvdimm);
> >> >  
> >> > @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
> >> >  		*cmd_rc = papr_scm_meta_set(p, buf);
> >> >  		break;
> >> >  
> >> > +	case ND_CMD_CALL:
> >> > +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
> >> > +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
> >> > +		break;
> >> > +
> >> >  	default:
> >> > -		return -EINVAL;
> >> > +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
> >> > +		*cmd_rc = -EINVAL;
> >> 
> >> Is this change related?  If there is a bug where there is a caller of
> >> papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
> >> that issue.
> >> 
> >> Ira
> >> 
> >> >  	}
> >> >  
> >> >  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
> >> >  
> >> > -	return 0;
> >> > +	return *cmd_rc;
> >> >  }
> >> >  
> >> >  static ssize_t flags_show(struct device *dev,
> >> > diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
> >> > index de5d90212409..0e09dc5cec19 100644
> >> > --- a/include/uapi/linux/ndctl.h
> >> > +++ b/include/uapi/linux/ndctl.h
> >> > @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
> >> >  #define NVDIMM_FAMILY_HPE2 2
> >> >  #define NVDIMM_FAMILY_MSFT 3
> >> >  #define NVDIMM_FAMILY_HYPERV 4
> >> > +#define NVDIMM_FAMILY_PAPR 5
> >> >  
> >> >  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
> >> >  					struct nd_cmd_pkg)
> >> > -- 
> >> > 2.26.2
> >> > 
> >> _______________________________________________
> >> Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
> >> To unsubscribe send an email to linux-nvdimm-leave@lists.01.org
> 
> -- 
> Cheers
> ~ Vaibhav

[ forgive formatting I'm temporarily stuck using Outlook this week... ]

> From: Vaibhav Jain <vaibhav@linux.ibm.com>
[..]
> 
> Introduce support for PAPR NVDIMM Specific Methods (PDSM) in papr_scm
> module and add the command family NVDIMM_FAMILY_PAPR to the white
> list of NVDIMM command sets. Also advertise support for ND_CMD_CALL for
> the nvdimm command mask and implement necessary scaffolding in the
> module to handle ND_CMD_CALL ioctl and PDSM requests that we receive.
> 
> The layout of the PDSM request as we expect from libnvdimm/libndctl is
> described in newly introduced uapi header 'papr_pdsm.h' which defines a
> new 'struct nd_pdsm_cmd_pkg' header. This header is used to communicate
> the PDSM request via member 'nd_cmd_pkg.nd_command' and size of
> payload that need to be sent/received for servicing the PDSM.
> 
> A new function is_cmd_valid() is implemented that reads the args to
> papr_scm_ndctl() and performs sanity tests on them. A new function
> papr_scm_service_pdsm() is introduced and is called from
> papr_scm_ndctl() in case of a PDSM request is received via ND_CMD_CALL
> command from libnvdimm.
> 
> Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Ira Weiny <ira.weiny@intel.com>
> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
> Changelog:
> 
> Resend:
> * Added ack from Aneesh.
> 
> v8..v9:
> * Reduced the usage of term SCM replacing it with appropriate
>   replacement [ Dan Williams, Aneesh ]
> * Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
> * s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
> * s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
> * Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
> * Minor update to patch description.
> 
> v7..v8:
> * Removed the 'payload_offset' field from 'struct
>   nd_pdsm_cmd_pkg'. Instead command payload is always assumed to start
>   at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
> * To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
>   'reserved' field of 10-bytes is introduced. [ Aneesh ]
> * Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
>   [ Ira ]
> 
> Resend:
> * None
> 
> v6..v7 :
> * Removed the re-definitions of __packed macro from papr_scm_pdsm.h
>   [Mpe].
> * Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h [Mpe].
> * Removed macros that were unused in papr_scm.c from papr_scm_pdsm.h
>   [Mpe].
> * Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]
> 
> v5..v6 :
> * Changed the usage of the term DSM to PDSM to distinguish it from the
>   ACPI term [ Dan Williams ]
> * Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various
> struct
>   to reflect the new terminology.
> * Updated the patch description and title to reflect the new terminology.
> * Squashed patch to introduce new command family in 'ndctl.h' with
>   this patch [ Dan Williams ]
> * Updated the papr_scm_pdsm method starting index from 0x10000 to 0x0
>   [ Dan Williams ]
> * Removed redundant license text from the papr_scm_psdm.h file.
>   [ Dan Williams ]
> * s/envelop/envelope/ at various places [ Dan Williams ]
> * Added '__packed' attribute to command package header to gaurd
>   against different compiler adding paddings between the fields.
>   [ Dan Williams]
> * Converted various pr_debug to dev_debug [ Dan Williams ]
> 
> v4..v5 :
> * None
> 
> v3..v4 :
> * None
> 
> v2..v3 :
> * Updated the patch prefix to 'ndctl/uapi' [Aneesh]
> 
> v1..v2 :
> * None
> ---
>  arch/powerpc/include/uapi/asm/papr_pdsm.h | 136
> ++++++++++++++++++++++  arch/powerpc/platforms/pseries/papr_scm.c |
> 101 +++++++++++++++-
>  include/uapi/linux/ndctl.h                |   1 +
>  3 files changed, 232 insertions(+), 6 deletions(-)  create mode 100644
> arch/powerpc/include/uapi/asm/papr_pdsm.h
> 
> diff --git a/arch/powerpc/include/uapi/asm/papr_pdsm.h
> b/arch/powerpc/include/uapi/asm/papr_pdsm.h
> new file mode 100644
> index 000000000000..6407fefcc007
> --- /dev/null
> +++ b/arch/powerpc/include/uapi/asm/papr_pdsm.h
> @@ -0,0 +1,136 @@
> +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
> +/*
> + * PAPR nvDimm Specific Methods (PDSM) and structs for libndctl
> + *
> + * (C) Copyright IBM 2020
> + *
> + * Author: Vaibhav Jain <vaibhav at linux.ibm.com>  */
> +
> +#ifndef _UAPI_ASM_POWERPC_PAPR_PDSM_H_
> +#define _UAPI_ASM_POWERPC_PAPR_PDSM_H_
> +
> +#include <linux/types.h>
> +
> +/*
> + * PDSM Envelope:
> + *
> + * The ioctl ND_CMD_CALL transfers data between user-space and kernel
> +via
> + * envelope which consists of a header and user-defined payload sections.
> + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
> + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
> + * There is reserved field that can used to introduce new fields to the
> + * structure in future. It also tries to ensure that
> 'nd_pdsm_cmd_pkg.payload'
> + * lies at a 8-byte boundary.
> + *
> + *  +-------------+---------------------+---------------------------+
> + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
> + *  +-------------+---------------------+---------------------------+
> + *  |               nd_pdsm_cmd_pkg     |                           |
> + *  |-------------+                     |                           |
> + *  |  nd_cmd_pkg |                     |                           |
> + *  +-------------+---------------------+---------------------------+
> + *  | nd_family   |                     |                           |
> + *  | nd_size_out | cmd_status          |                           |
> + *  | nd_size_in  | payload_version     |     payload               |
> + *  | nd_command  | reserved            |                           |
> + *  | nd_fw_size  |                     |                           |
> + *  +-------------+---------------------+---------------------------+
> + *
> + * PDSM Header:
> + *
> + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
> + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to
> member
> + * 'nd_cmd_pkg.nd_command'. Apart from size information of the
> envelope
> +which is
> + * contained in 'struct nd_cmd_pkg', the header also has members
> +following
> + * members:
> + *
> + * 'cmd_status'		: (Out) Errors if any encountered while
> servicing PDSM.
> + * 'payload_version'	: (In/Out) Version number associated with the
> payload.
> + * 'reserved'		: Not used and reserved for future.
> + *
> + * PDSM Payload:
> + *
> + * The layout of the PDSM Payload is defined by various structs shared
> +between
> + * papr_scm and libndctl so that contents of payload can be
> +interpreted. During
> + * servicing of a PDSM the papr_scm module will read input args from
> +the payload
> + * field by casting its contents to an appropriate struct pointer based
> +on the
> + * PDSM command. Similarly the output of servicing the PDSM command
> +will be
> + * copied to the payload field using the same struct.
> + *
> + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size,
> +which
> + * leaves around 176 bytes for the envelope payload (ignoring any
> +padding that
> + * the compiler may silently introduce).
> + *
> + * Payload Version:
> + *
> + * A 'payload_version' field is present in PDSM header that indicates a
> +specific
> + * version of the structure present in PDSM Payload for a given PDSM
> command.
> + * This provides backward compatibility in case the PDSM Payload
> +structure
> + * evolves and different structures are supported by 'papr_scm' and
> 'libndctl'.
> + *
> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send
> +the version
> + * of the payload struct it supports via 'payload_version' field. The
> 'papr_scm'
> + * module when servicing the PDSM envelope checks the 'payload_version'
> +and then
> + * uses 'payload struct version' == MIN('payload_version field',
> + * 'max payload-struct-version supported by papr_scm') to service the
> PDSM.
> + * After servicing the PDSM, 'papr_scm' put the negotiated version of
> +payload
> + * struct in returned 'payload_version' field.
> + *
> + * Libndctl on receiving the envelope back from papr_scm again checks
> +the
> + * 'payload_version' field and based on it use the appropriate version
> +dsm
> + * struct to parse the results.
> + *
> + * Backward Compatibility:
> + *
> + * Above scheme of exchanging different versioned PDSM struct between
> +libndctl
> + * and papr_scm should provide backward compatibility until following
> +two
> + * assumptions/conditions when defining new PDSM structs hold:
> + *
> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
> + *
> + * 1. T(X) is a proper subset of T(Y) if Y > X.
> + *    i.e Each new version of PDSM struct should retain existing struct
> + *    attributes from previous version
> + *
> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
> + *    it should also support T(1), T(2)...T(X - 1).
> + *    i.e When adding support for new version of a PDSM struct, libndctl
> + *    and papr_scm should retain support of the existing PDSM struct
> + *    version they support.
> + */
> +
> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from
> libnvdimm
> +*/ struct nd_pdsm_cmd_pkg {
> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-
> cmd */
> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
> +	__u16 reserved[5];	/* Ignored and to be used in future */
> +	__u16 payload_version;	/* In/Out: version of the payload */
> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
> +} __packed;
> +
> +/*
> + * Methods to be embedded in ND_CMD_CALL request. These are sent to
> the
> +kernel
> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct  */
> +enum papr_pdsm {
> +	PAPR_PDSM_MIN = 0x0,
> +	PAPR_PDSM_MAX,
> +};
> +
> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */ static inline
> +struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg
> *cmd) {
> +	return (struct nd_pdsm_cmd_pkg *) cmd; }
> +
> +/* Return the payload pointer for a given pcmd */ static inline void
> +*pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd) {
> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
> +		return NULL;
> +	else
> +		return (void *)(pcmd->payload);
> +}
> +
> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c
> b/arch/powerpc/platforms/pseries/papr_scm.c
> index 149431594839..5e2237e7ec08 100644
> --- a/arch/powerpc/platforms/pseries/papr_scm.c
> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> @@ -15,13 +15,15 @@
>  #include <linux/seq_buf.h>
> 
>  #include <asm/plpar_wrappers.h>
> +#include <asm/papr_pdsm.h>
> 
>  #define BIND_ANY_ADDR (~0ul)
> 
>  #define PAPR_SCM_DIMM_CMD_MASK \
>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
> +	 (1ul << ND_CMD_CALL))
> 
>  /* DIMM health bitmap bitmap indicators */
>  /* SCM device is unable to persist memory contents */ @@ -350,16 +352,97
> @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>  	return 0;
>  }
> 
> +/*
> + * Validate the inputs args to dimm-control function and return '0' if valid.
> + * This also does initial sanity validation to ND_CMD_CALL sub-command
> packages.
> + */
> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void
> *buf,
> +		       unsigned int buf_len)
> +{
> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
> +	struct papr_scm_priv *p;
> +
> +	/* Only dimm-specific calls are supported atm */
> +	if (!nvdimm)
> +		return -EINVAL;
> +
> +	/* get the provider date from struct nvdimm */
> +	p = nvdimm_provider_data(nvdimm);
> +
> +	if (!test_bit(cmd, &cmd_mask)) {
> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
> +		return -EINVAL;
> +	} else if (cmd == ND_CMD_CALL) {
> +
> +		/* Verify the envelope package */
> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
> +				buf_len);
> +			return -EINVAL;
> +		}
> +
> +		/* Verify that the PDSM family is valid */
> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
> +			dev_dbg(&p->pdev->dev, "Invalid pkg
> family=0x%llx\n",
> +				pkg->hdr.nd_family);
> +			return -EINVAL;
> +
> +		}
> +
> +		/* We except a payload with all PDSM commands */
> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
> +			dev_dbg(&p->pdev->dev,
> +				"Empty payload for sub-command=0x%llx\n",
> +				pkg->hdr.nd_command);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	/* Command looks valid */

So this is where I would expect the kernel to validate the command vs a known list of supported commands / payloads. One of the goals of requiring public documentation of any commands that libnvdimm might support for the ioctl path is to give the kernel the ability to gate future enabling on consideration of a common kernel front-end interface. I believe this would also address questions about the versioning scheme because userspace would be actively prevented from sending command payloads that were not first explicitly enabled in the kernel. This interface as it stands in this patch set seems to be a very thin / "anything goes" passthrough with no consideration for that policy.

As an example of the utility of this policy, consider the recent support for nvdimm security commands that allow a passphrase to be set and issue commands like "unlock" and "secure erase". The kernel actively prevents those commands from being sent from userspace. See acpi_nfit_clear_to_send() and nd_cmd_clear_to_send(). The reasoning is that it enforces the kernel's nvdimm security model that uses encrypted/trusted keys to protect key material (clear text keys only-ever exist in kernel-space). Yes, that restriction is painful for people that don't want the kernel's security model and just want the simplicity of passing clear-text keys around, but it's necessary for the kernel to have any chance to provide a common abstraction across vendors. The pain of negotiating every single command with what the kernel will support is useful for the long term health of the kernel. It forces ongoing conversations across vendors to consolidate interfaces and reuse kernel best practices like encrypted/trusted keys. Code acceptance is the only real gate the kernel has to enforce cooperation across vendors.

The expectation is that the kernel does not allow any command to pass that is not explicitly listed in a bitmap of known commands. I would expect that if you changed the payload of an existing command that would likely require a new entry in this bitmap. The goal is to give the kernel a chance to constrain the passthrough interface to afford a chance to have a discussion of what might done in a common implementation. Another example is the label-area read-write commands. The kernel needs explicit control to ensure that it owns the label area and that userspace is not able to corrupt it (write it behind the kernel's back).

Now that said, I have battle scars with some OEMs that just want a generic passthrough interface so they never need to work with the kernel community again and can just write their custom validation tooling and be done. I've mostly been successful in that fight outside of the gaping hole of ND_CMD_VENDOR. That's the path that ipmctl has used to issue commands that have not made it into the public specification on docs.pmem.io. My warning shot for that is the "disable_vendor_specific" module option that administrators can set to only allow commands that the kernel explicitly knows the effects of to be issued. The result is only tooling / enabling that submits to this auditing regime is guaranteed to work everywhere.

So, that long explanation out of the way, what does that mean for this patch set? I'd like to understand if you still see a need for a versioning scheme if the implementation is required to explicitly list all the commands it supports? I.e. that the kernel need not worry about userspace sending future unknown payloads because unknown payloads are blocked. Also if your interface has anything similar to a "vendor specific" passthrough I would like to require that go through the ND_CMD_VENDOR ioctl, so that the kernel still has a common check point to prevent vendor specific "I don't want to talk to the kernel community" shenanigans, but even better if ND_CMD_VENDOR is something the kernel can eventually jettison because nobody is using it.

I feel like this is a conversation that will take a few days to resolve, which does not leave time to push this for v5.8. That said, I do think the health flags patches at the beginning of this series are low risk and uncontentious. How about I merge those for v5.8 and circle back to get this ioctl path queued early in v5.8-rc? Apologies for the late feedback on this relative to v5.8.

Hi Dan,

Thanks for review and insights on this. My responses below:

"Williams, Dan J" <dan.j.williams@intel.com> writes:

> [ forgive formatting I'm temporarily stuck using Outlook this week... ]
>
>> From: Vaibhav Jain <vaibhav@linux.ibm.com>
> [..]
>> 
>> Introduce support for PAPR NVDIMM Specific Methods (PDSM) in papr_scm
>> module and add the command family NVDIMM_FAMILY_PAPR to the white
>> list of NVDIMM command sets. Also advertise support for ND_CMD_CALL for
>> the nvdimm command mask and implement necessary scaffolding in the
>> module to handle ND_CMD_CALL ioctl and PDSM requests that we receive.
>> 
>> The layout of the PDSM request as we expect from libnvdimm/libndctl is
>> described in newly introduced uapi header 'papr_pdsm.h' which defines a
>> new 'struct nd_pdsm_cmd_pkg' header. This header is used to communicate
>> the PDSM request via member 'nd_cmd_pkg.nd_command' and size of
>> payload that need to be sent/received for servicing the PDSM.
>> 
>> A new function is_cmd_valid() is implemented that reads the args to
>> papr_scm_ndctl() and performs sanity tests on them. A new function
>> papr_scm_service_pdsm() is introduced and is called from
>> papr_scm_ndctl() in case of a PDSM request is received via ND_CMD_CALL
>> command from libnvdimm.
>> 
>> Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
>> Cc: Dan Williams <dan.j.williams@intel.com>
>> Cc: Michael Ellerman <mpe@ellerman.id.au>
>> Cc: Ira Weiny <ira.weiny@intel.com>
>> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
>> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
>> ---
>> Changelog:
>> 
>> Resend:
>> * Added ack from Aneesh.
>> 
>> v8..v9:
>> * Reduced the usage of term SCM replacing it with appropriate
>>   replacement [ Dan Williams, Aneesh ]
>> * Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
>> * s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
>> * s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
>> * Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
>> * Minor update to patch description.
>> 
>> v7..v8:
>> * Removed the 'payload_offset' field from 'struct
>>   nd_pdsm_cmd_pkg'. Instead command payload is always assumed to start
>>   at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
>> * To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
>>   'reserved' field of 10-bytes is introduced. [ Aneesh ]
>> * Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
>>   [ Ira ]
>> 
>> Resend:
>> * None
>> 
>> v6..v7 :
>> * Removed the re-definitions of __packed macro from papr_scm_pdsm.h
>>   [Mpe].
>> * Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h [Mpe].
>> * Removed macros that were unused in papr_scm.c from papr_scm_pdsm.h
>>   [Mpe].
>> * Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]
>> 
>> v5..v6 :
>> * Changed the usage of the term DSM to PDSM to distinguish it from the
>>   ACPI term [ Dan Williams ]
>> * Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various
>> struct
>>   to reflect the new terminology.
>> * Updated the patch description and title to reflect the new terminology.
>> * Squashed patch to introduce new command family in 'ndctl.h' with
>>   this patch [ Dan Williams ]
>> * Updated the papr_scm_pdsm method starting index from 0x10000 to 0x0
>>   [ Dan Williams ]
>> * Removed redundant license text from the papr_scm_psdm.h file.
>>   [ Dan Williams ]
>> * s/envelop/envelope/ at various places [ Dan Williams ]
>> * Added '__packed' attribute to command package header to gaurd
>>   against different compiler adding paddings between the fields.
>>   [ Dan Williams]
>> * Converted various pr_debug to dev_debug [ Dan Williams ]
>> 
>> v4..v5 :
>> * None
>> 
>> v3..v4 :
>> * None
>> 
>> v2..v3 :
>> * Updated the patch prefix to 'ndctl/uapi' [Aneesh]
>> 
>> v1..v2 :
>> * None
>> ---
>>  arch/powerpc/include/uapi/asm/papr_pdsm.h | 136
>> ++++++++++++++++++++++  arch/powerpc/platforms/pseries/papr_scm.c |
>> 101 +++++++++++++++-
>>  include/uapi/linux/ndctl.h                |   1 +
>>  3 files changed, 232 insertions(+), 6 deletions(-)  create mode 100644
>> arch/powerpc/include/uapi/asm/papr_pdsm.h
>> 
>> diff --git a/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> b/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> new file mode 100644
>> index 000000000000..6407fefcc007
>> --- /dev/null
>> +++ b/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> @@ -0,0 +1,136 @@
>> +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
>> +/*
>> + * PAPR nvDimm Specific Methods (PDSM) and structs for libndctl
>> + *
>> + * (C) Copyright IBM 2020
>> + *
>> + * Author: Vaibhav Jain <vaibhav at linux.ibm.com>  */
>> +
>> +#ifndef _UAPI_ASM_POWERPC_PAPR_PDSM_H_
>> +#define _UAPI_ASM_POWERPC_PAPR_PDSM_H_
>> +
>> +#include <linux/types.h>
>> +
>> +/*
>> + * PDSM Envelope:
>> + *
>> + * The ioctl ND_CMD_CALL transfers data between user-space and kernel
>> +via
>> + * envelope which consists of a header and user-defined payload sections.
>> + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects a
>> + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload' field.
>> + * There is reserved field that can used to introduce new fields to the
>> + * structure in future. It also tries to ensure that
>> 'nd_pdsm_cmd_pkg.payload'
>> + * lies at a 8-byte boundary.
>> + *
>> + *  +-------------+---------------------+---------------------------+
>> + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
>> + *  +-------------+---------------------+---------------------------+
>> + *  |               nd_pdsm_cmd_pkg     |                           |
>> + *  |-------------+                     |                           |
>> + *  |  nd_cmd_pkg |                     |                           |
>> + *  +-------------+---------------------+---------------------------+
>> + *  | nd_family   |                     |                           |
>> + *  | nd_size_out | cmd_status          |                           |
>> + *  | nd_size_in  | payload_version     |     payload               |
>> + *  | nd_command  | reserved            |                           |
>> + *  | nd_fw_size  |                     |                           |
>> + *  +-------------+---------------------+---------------------------+
>> + *
>> + * PDSM Header:
>> + *
>> + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
>> + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to
>> member
>> + * 'nd_cmd_pkg.nd_command'. Apart from size information of the
>> envelope
>> +which is
>> + * contained in 'struct nd_cmd_pkg', the header also has members
>> +following
>> + * members:
>> + *
>> + * 'cmd_status'		: (Out) Errors if any encountered while
>> servicing PDSM.
>> + * 'payload_version'	: (In/Out) Version number associated with the
>> payload.
>> + * 'reserved'		: Not used and reserved for future.
>> + *
>> + * PDSM Payload:
>> + *
>> + * The layout of the PDSM Payload is defined by various structs shared
>> +between
>> + * papr_scm and libndctl so that contents of payload can be
>> +interpreted. During
>> + * servicing of a PDSM the papr_scm module will read input args from
>> +the payload
>> + * field by casting its contents to an appropriate struct pointer based
>> +on the
>> + * PDSM command. Similarly the output of servicing the PDSM command
>> +will be
>> + * copied to the payload field using the same struct.
>> + *
>> + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size,
>> +which
>> + * leaves around 176 bytes for the envelope payload (ignoring any
>> +padding that
>> + * the compiler may silently introduce).
>> + *
>> + * Payload Version:
>> + *
>> + * A 'payload_version' field is present in PDSM header that indicates a
>> +specific
>> + * version of the structure present in PDSM Payload for a given PDSM
>> command.
>> + * This provides backward compatibility in case the PDSM Payload
>> +structure
>> + * evolves and different structures are supported by 'papr_scm' and
>> 'libndctl'.
>> + *
>> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send
>> +the version
>> + * of the payload struct it supports via 'payload_version' field. The
>> 'papr_scm'
>> + * module when servicing the PDSM envelope checks the 'payload_version'
>> +and then
>> + * uses 'payload struct version' == MIN('payload_version field',
>> + * 'max payload-struct-version supported by papr_scm') to service the
>> PDSM.
>> + * After servicing the PDSM, 'papr_scm' put the negotiated version of
>> +payload
>> + * struct in returned 'payload_version' field.
>> + *
>> + * Libndctl on receiving the envelope back from papr_scm again checks
>> +the
>> + * 'payload_version' field and based on it use the appropriate version
>> +dsm
>> + * struct to parse the results.
>> + *
>> + * Backward Compatibility:
>> + *
>> + * Above scheme of exchanging different versioned PDSM struct between
>> +libndctl
>> + * and papr_scm should provide backward compatibility until following
>> +two
>> + * assumptions/conditions when defining new PDSM structs hold:
>> + *
>> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
>> + *
>> + * 1. T(X) is a proper subset of T(Y) if Y > X.
>> + *    i.e Each new version of PDSM struct should retain existing struct
>> + *    attributes from previous version
>> + *
>> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
>> + *    it should also support T(1), T(2)...T(X - 1).
>> + *    i.e When adding support for new version of a PDSM struct, libndctl
>> + *    and papr_scm should retain support of the existing PDSM struct
>> + *    version they support.
>> + */
>> +
>> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from
>> libnvdimm
>> +*/ struct nd_pdsm_cmd_pkg {
>> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-
>> cmd */
>> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
>> +	__u16 reserved[5];	/* Ignored and to be used in future */
>> +	__u16 payload_version;	/* In/Out: version of the payload */
>> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
>> +} __packed;
>> +
>> +/*
>> + * Methods to be embedded in ND_CMD_CALL request. These are sent to
>> the
>> +kernel
>> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct  */
>> +enum papr_pdsm {
>> +	PAPR_PDSM_MIN = 0x0,
>> +	PAPR_PDSM_MAX,
>> +};
>> +
>> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */ static inline
>> +struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg
>> *cmd) {
>> +	return (struct nd_pdsm_cmd_pkg *) cmd; }
>> +
>> +/* Return the payload pointer for a given pcmd */ static inline void
>> +*pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd) {
>> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
>> +		return NULL;
>> +	else
>> +		return (void *)(pcmd->payload);
>> +}
>> +
>> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
>> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c
>> b/arch/powerpc/platforms/pseries/papr_scm.c
>> index 149431594839..5e2237e7ec08 100644
>> --- a/arch/powerpc/platforms/pseries/papr_scm.c
>> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
>> @@ -15,13 +15,15 @@
>>  #include <linux/seq_buf.h>
>> 
>>  #include <asm/plpar_wrappers.h>
>> +#include <asm/papr_pdsm.h>
>> 
>>  #define BIND_ANY_ADDR (~0ul)
>> 
>>  #define PAPR_SCM_DIMM_CMD_MASK \
>>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
>> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
>> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
>> +	 (1ul << ND_CMD_CALL))
>> 
>>  /* DIMM health bitmap bitmap indicators */
>>  /* SCM device is unable to persist memory contents */ @@ -350,16 +352,97
>> @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>>  	return 0;
>>  }
>> 
>> +/*
>> + * Validate the inputs args to dimm-control function and return '0' if valid.
>> + * This also does initial sanity validation to ND_CMD_CALL sub-command
>> packages.
>> + */
>> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void
>> *buf,
>> +		       unsigned int buf_len)
>> +{
>> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
>> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
>> +	struct papr_scm_priv *p;
>> +
>> +	/* Only dimm-specific calls are supported atm */
>> +	if (!nvdimm)
>> +		return -EINVAL;
>> +
>> +	/* get the provider date from struct nvdimm */
>> +	p = nvdimm_provider_data(nvdimm);
>> +
>> +	if (!test_bit(cmd, &cmd_mask)) {
>> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
>> +		return -EINVAL;
>> +	} else if (cmd == ND_CMD_CALL) {
>> +
>> +		/* Verify the envelope package */
>> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
>> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
>> +				buf_len);
>> +			return -EINVAL;
>> +		}
>> +
>> +		/* Verify that the PDSM family is valid */
>> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
>> +			dev_dbg(&p->pdev->dev, "Invalid pkg
>> family=0x%llx\n",
>> +				pkg->hdr.nd_family);
>> +			return -EINVAL;
>> +
>> +		}
>> +
>> +		/* We except a payload with all PDSM commands */
>> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
>> +			dev_dbg(&p->pdev->dev,
>> +				"Empty payload for sub-command=0x%llx\n",
>> +				pkg->hdr.nd_command);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	/* Command looks valid */
>
<snip>
> So this is where I would expect the kernel to validate the command vs
> a known list of supported commands / payloads. One of the goals of
> requiring public documentation of any commands that libnvdimm might
> support for the ioctl path is to give the kernel the ability to gate
> future enabling on consideration of a common kernel front-end
> interface. I believe this would also address questions about the
> versioning scheme because userspace would be actively prevented from
> sending command payloads that were not first explicitly enabled in the
> kernel. This interface as it stands in this patch set seems to be a
> very thin / "anything goes" passthrough with no consideration for that
> policy.
>
> As an example of the utility of this policy, consider the recent
> support for nvdimm security commands that allow a passphrase to be set
> and issue commands like "unlock" and "secure erase". The kernel
> actively prevents those commands from being sent from userspace. See
> acpi_nfit_clear_to_send() and nd_cmd_clear_to_send(). The reasoning is
> that it enforces the kernel's nvdimm security model that uses
> encrypted/trusted keys to protect key material (clear text keys
> only-ever exist in kernel-space). Yes, that restriction is painful for
> people that don't want the kernel's security model and just want the
> simplicity of passing clear-text keys around, but it's necessary for
> the kernel to have any chance to provide a common abstraction across
> vendors. The pain of negotiating every single command with what the
> kernel will support is useful for the long term health of the
> kernel. It forces ongoing conversations across vendors to consolidate
> interfaces and reuse kernel best practices like encrypted/trusted
> keys. Code acceptance is the only real gate the kernel has to enforce
> cooperation across vendors.
>
> The expectation is that the kernel does not allow any command to pass
> that is not explicitly listed in a bitmap of known commands. I would
> expect that if you changed the payload of an existing command that
> would likely require a new entry in this bitmap. The goal is to give
> the kernel a chance to constrain the passthrough interface to afford a
> chance to have a discussion of what might done in a common
> implementation. Another example is the label-area read-write
> commands. The kernel needs explicit control to ensure that it owns the
> label area and that userspace is not able to corrupt it (write it
> behind the kernel's back).
>
> Now that said, I have battle scars with some OEMs that just want a
> generic passthrough interface so they never need to work with the
> kernel community again and can just write their custom validation
> tooling and be done. I've mostly been successful in that fight outside
> of the gaping hole of ND_CMD_VENDOR. That's the path that ipmctl has
> used to issue commands that have not made it into the public
> specification on docs.pmem.io. My warning shot for that is the
> "disable_vendor_specific" module option that administrators can set to
> only allow commands that the kernel explicitly knows the effects of to
> be issued. The result is only tooling / enabling that submits to this
> auditing regime is guaranteed to work everywhere.

Agree with points made above. With this patchset we arent really trying
to push an ioctl passthrough to exchange arbitary data with
papr-scm module. Nor do we want to bypass the kernel community for any
future enhancements on this interface. We made some design choices based on
our understanding of certain restriction we saw in
ndctl/libndctl. Specifically wanted to avoid issuing two CMD_CALL ioctl
roundtrips.

That being said I had an extended discussion with Aneesh rethinking the
'version' field and we both agreed *to remove this field* from the
proposed 'struct nd_pdsm_cmd_pkg'. This should resolve the contentions
around this Patch-4 in this patchset. Since the 'version' field isnt
extensively used right now the impact on the patchset would be small.

>
> So, that long explanation out of the way, what does that mean for this
> patch set? I'd like to understand if you still see a need for a
> versioning scheme if the implementation is required to explicitly list
> all the commands it supports? I.e. that the kernel need not worry
> about userspace sending future unknown payloads because unknown
> payloads are blocked. Also if your interface has anything similar to a
> "vendor specific" passthrough I would like to require that go through
> the ND_CMD_VENDOR ioctl, so that the kernel still has a common check
> point to prevent vendor specific "I don't want to talk to the kernel
> community" shenanigans, but even better if ND_CMD_VENDOR is something
> the kernel can eventually jettison because nobody is using it.

As I mentioned above this isn't a 'vendor specific passthrough'
machenism. The 'version' field was proposed to avoid two CMD_CALL ioctl
roundtrip to fetch and report extended nvdimm health data like
'life-remaining' which isnt always available for papr-scm.

However we just realized instead of relying on 'version' field we can
advertise support for these extended attributes via nvdimm-flags from
sysfs. Looking at the nvdimm-flags libndctl can use an appropriate
pdsm command and struct to fetch the dimm health information from
papr_scm via CMD_CALL.

But thats something we plan to do in future and not with the current
patchset which only reports fixed set of nvdimm health attributes.

>
> I feel like this is a conversation that will take a few days to
> resolve, which does not leave time to push this for v5.8. That said, I
> do think the health flags patches at the beginning of this series are
> low risk and uncontentious. How about I merge those for v5.8 and
> circle back to get this ioctl path queued early in v5.8-rc? Apologies
> for the late feedback on this relative to v5.8.
>
Thanks for this consideration. Agree to the proposal. However changes to
patchset with removal of 'version' field is fairly small hence can
quickly push an updated patch series cumulating rest of the review
comments from Ira.

Does that sounds reasonable ?

Thanks,
~ Vaibhav

Ira Weiny <ira.weiny@intel.com> writes:

> On Wed, Jun 03, 2020 at 11:41:42PM +0530, Vaibhav Jain wrote:
>> Hi Ira,
>> 
>> Thanks for reviewing this patch. My responses below:
>> 
>> Ira Weiny <ira.weiny@intel.com> writes:
>> 
>
> ...
>
>> >> + *
>> >> + * Payload Version:
>> >> + *
>> >> + * A 'payload_version' field is present in PDSM header that indicates a specific
>> >> + * version of the structure present in PDSM Payload for a given PDSM command.
>> >> + * This provides backward compatibility in case the PDSM Payload structure
>> >> + * evolves and different structures are supported by 'papr_scm' and 'libndctl'.
>> >> + *
>> >> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send the version
>> >> + * of the payload struct it supports via 'payload_version' field. The 'papr_scm'
>> >> + * module when servicing the PDSM envelope checks the 'payload_version' and then
>> >> + * uses 'payload struct version' == MIN('payload_version field',
>> >> + * 'max payload-struct-version supported by papr_scm') to service the PDSM.
>> >> + * After servicing the PDSM, 'papr_scm' put the negotiated version of payload
>> >> + * struct in returned 'payload_version' field.
>> >> + *
>> >> + * Libndctl on receiving the envelope back from papr_scm again checks the
>> >> + * 'payload_version' field and based on it use the appropriate version dsm
>> >> + * struct to parse the results.
>> >> + *
>> >> + * Backward Compatibility:
>> >> + *
>> >> + * Above scheme of exchanging different versioned PDSM struct between libndctl
>> >> + * and papr_scm should provide backward compatibility until following two
>> >> + * assumptions/conditions when defining new PDSM structs hold:
>> >> + *
>> >> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
>> >> + *
>> >> + * 1. T(X) is a proper subset of T(Y) if Y > X.
>> >> + *    i.e Each new version of PDSM struct should retain existing struct
>> >> + *    attributes from previous version
>> >> + *
>> >> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
>> >> + *    it should also support T(1), T(2)...T(X - 1).
>> >> + *    i.e When adding support for new version of a PDSM struct, libndctl
>> >> + *    and papr_scm should retain support of the existing PDSM struct
>> >> + *    version they support.
>> >
>> > Please see this thread for an example why versions are a bad idea in UAPIs:
>> >
>> > https://lkml.org/lkml/2020/3/26/213
>> >
>> 
>> > While the use of version is different in that thread the fundamental issues are
>> > the same.  You end up with some weird matrix of supported features and
>> > structure definitions.  For example, you are opening up the possibility of
>> > changing structures with a different version for no good reason.
>> 
>> Not really sure I understand the statement correctly "you are opening up
>> the possibility of changing structures with a different version for no
>> good reason."
>
[..]
> What I mean is:
>
> struct v1 {
> 	u32 x;
> 	u32 y;
> };
>
> struct v2 {
> 	u32 y;
> 	u32 x;
> };
>
> x and y are the same data but you have now redefined the order of the struct.
> You don't need that flexibility/complexity.
>
> Generally I think you are defining:
>
> struct v1 {
> 	u32 x;
> 	u32 y;
> };
>
> struct v2 {
> 	u32 x;
> 	u32 y;
> 	u32 z;
> 	u32 a;
> };
>
> Which becomes 2 structures...  There is no need.
>
> The easiest thing to do is:
>
> struct user_data {
> 	u32 x;
> 	u32 y;
> };
>
> And later you modify user_data to:
>
> struct user_data {
> 	u32 x;
> 	u32 y;
> 	u32 z;
> 	u32 a;
> };
>
> libndctl always passes sizeof(struct user_data) to the call. [Do ensure
> structures are 64bit aligned for this to work.]
>
> The kernel sees the size and returns the amount of data up to that size.
>
> Therefore, older kernels automatically fill in x and y,  newer kernels fill in
> z/a if the buffer was big enough.  libndctl only uses the fields it knows about.
>
> It is _much_ easier this way.  Almost nothing needs to get changed as versions
> roll forward.  The only big issue is if libndctl _needs_ z then it has to check
> if z is returned.
>
> In that case add a cap_mask with bit fields which the kernel can fill in for
> which fields are valid.
>
> struct user_data {
> 	u64 cap_mask;  /* where bits define extra future capabilities */
> 	u32 x;
> 	u32 y;
> };
>
> IFF you need to add data within fields which are reserved you can use
> capability flags to indicate which fields are requested and which are returned
> by the kernel.
>
> But I _think_ for what you want libndctl must survive if z/a are not available
> right?  So just adding to the structure should be fine.
Agreed. But as I mentioned in my response to Dan's review comments [1], we
will be removing the version field altogether and instead will introduce
new psdm requests bound to new struct definitions in conjuntion to
nvdimm-flags. I have a patchset ready which I will be sending out soon.

[1] https://lore.kernel.org/linux-nvdimm/87h7vrgpzx.fsf@linux.ibm.com/
>
>> We want to return more data in the struct in future iterations. So
>> 'changing structure with different version' is something we are
>> expecting. 
>> 
>> With the backward compatibility constraints 1 & 2 above, it will ensure
>> that support matrix looks like a lower traingular matrix with each
>> successive version supporting previous version attributes. So supporting
>> future versions is relatively simplified.
>
> But you end up with weird switch/if's etc to deal with the multiple structures.
>
> With the size method the kernel simply returns the same size data as the user
> requested and everything is done.  No logic required at all.  Literally it can
> just copy the data it has (truncating if necessary).
>
Agreed. But with version field gone now we will instead use new psdm
requests bound to new struct definitions in conjuntion to nvdimm-flags
to retrive extended data from papr_scm.

>> 
>> >
>> > Also having the user query with version Z and get back version X (older) is
>> > odd.  Generally if the kernel does not know about a feature (ie version Z of
>> > the structure) it should return -EINVAL and let the user figure out what to do.
>> > The user may just give up or they could try a different query.
>> >
>> Considering the flow of ndctl/libndctl this is needed. libndctl will
>> usually issues only one CMD_CALL ioctl to kernel and if that fails then
>> an error is reported and ndctl will exit loosing state.
>> 
>> Adding mechanism in libndctl to reissue CMD_CALL ioctl to fetch a
>> appropriate version of pdsm struct is going to be considerably more
>> work.
>> 
>> This version fall-back mechanism, ensures that libndctl will receive
>> usable data without having to reissue a more CMD_CALL ioctls.
>
> Define usable?
>
> What happens if libndctl does not get 'z' in my example above?  What does it
> do?  If I understand correctly it does not _need_ z.  So why have a check on
> the version from the kernel?
>
> What if we change to:
>
> struct v3 {
> 	u32 x;
> 	u32 y;
> 	u32 z;
> 	u32 a;
> 	u32 b;
> 	u32 c;
> };
>
> Now it has to
>
> 	if(version 2)
> 		z/a valid do something()
>
> 	if(version 3)
> 		b/c valid do something else()
>
> if z, a, b, c are all 0 does it matter?
>
> If not, the logic above disappears.
>
> If so, then you need a cap mask.  Then the kernel can say c and a are valid
> (but c is 0) or other flexible stuff like that.
>
>> 
>> >> + */
>> >> +
>> >> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */
>> >> +struct nd_pdsm_cmd_pkg {
>> >> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-cmd */
>> >> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
>> >> +	__u16 reserved[5];	/* Ignored and to be used in future */
>> >
>> > How do you know when reserved is used for something else in the future?  Is
>> > reserved guaranteed (and checked by the code) to be 0?
>> 
>> For current set of pdsm requests ignore these reserved fields. However a
>> future pdsm request can leverage these reserved fields. So papr_scm
>> just bind the usage of these fields with the value of
>> 'nd_cmd_pkg.nd_command' that indicates the pdsm request.
>> 
>> That being said checking if the reserved fields are set to 0 will be a
>> good measure. Will add this check in next iteration.
>
> Exactly, if you don't check them now you will end up with an older libndctl
> which passes in garbage and breaks future users...  Basically rendering the
> reserved fields useless.
I have addressed this in my new patch-series which adds checks for
reserved fields to be '0'

>
>> 
>> >
>> >> +	__u16 payload_version;	/* In/Out: version of the payload */
>> >
>> > Why is payload_version after reserved?
>> Want to place the payload version field just before the payload data so
>> that it can be accessed with simple pointer arithmetic.
>
> I did not see that in the patch.  I thought you were using
> nd_pdsm_cmd_pkg->payload_version?
Thats right, but it just provided another simple way to retrive
payload_version without resorting to 'container_of' macro. Anyways the
version field is now gone hence 'payload' follows the reserved fields.

>
>> 
>> >
>> >> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
>> >> +} __packed;
>> >> +
>> >> +/*
>> >> + * Methods to be embedded in ND_CMD_CALL request. These are sent to the kernel
>> >> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
>> >> + */
>> >> +enum papr_pdsm {
>> >> +	PAPR_PDSM_MIN = 0x0,
>> >> +	PAPR_PDSM_MAX,
>> >> +};
>> >> +
>> >> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */
>> >> +static inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct nd_cmd_pkg *cmd)
>> >> +{
>> >> +	return (struct nd_pdsm_cmd_pkg *) cmd;
>> >> +}
>> >> +
>> >> +/* Return the payload pointer for a given pcmd */
>> >> +static inline void *pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd)
>> >> +{
>> >> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
>> >> +		return NULL;
>> >> +	else
>> >> +		return (void *)(pcmd->payload);
>> >> +}
>> >> +
>> >> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
>> >> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
>> >> index 149431594839..5e2237e7ec08 100644
>> >> --- a/arch/powerpc/platforms/pseries/papr_scm.c
>> >> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
>> >> @@ -15,13 +15,15 @@
>> >>  #include <linux/seq_buf.h>
>> >>  
>> >>  #include <asm/plpar_wrappers.h>
>> >> +#include <asm/papr_pdsm.h>
>> >>  
>> >>  #define BIND_ANY_ADDR (~0ul)
>> >>  
>> >>  #define PAPR_SCM_DIMM_CMD_MASK \
>> >>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>> >>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
>> >> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
>> >> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
>> >> +	 (1ul << ND_CMD_CALL))
>> >>  
>> >>  /* DIMM health bitmap bitmap indicators */
>> >>  /* SCM device is unable to persist memory contents */
>> >> @@ -350,16 +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>> >>  	return 0;
>> >>  }
>> >>  
>> >> +/*
>> >> + * Validate the inputs args to dimm-control function and return '0' if valid.
>> >> + * This also does initial sanity validation to ND_CMD_CALL sub-command packages.
>> >> + */
>> >> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>> >> +		       unsigned int buf_len)
>> >> +{
>> >> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
>> >> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
>> >> +	struct papr_scm_priv *p;
>> >> +
>> >> +	/* Only dimm-specific calls are supported atm */
>> >> +	if (!nvdimm)
>> >> +		return -EINVAL;
>> >> +
>> >> +	/* get the provider date from struct nvdimm */
>> >
>> > s/date/data
>> Thanks for point this out. Will fix this in next iteration.
>> 
>> >
>> >> +	p = nvdimm_provider_data(nvdimm);
>> >> +
>> >> +	if (!test_bit(cmd, &cmd_mask)) {
>> >> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
>> >> +		return -EINVAL;
>> >> +	} else if (cmd == ND_CMD_CALL) {
>> >> +
>> >> +		/* Verify the envelope package */
>> >> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
>> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
>> >> +				buf_len);
>> >> +			return -EINVAL;
>> >> +		}
>> >> +
>> >> +		/* Verify that the PDSM family is valid */
>> >> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
>> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg family=0x%llx\n",
>> >> +				pkg->hdr.nd_family);
>> >> +			return -EINVAL;
>> >> +
>> >> +		}
>> >> +
>> >> +		/* We except a payload with all PDSM commands */
>> >> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
>> >> +			dev_dbg(&p->pdev->dev,
>> >> +				"Empty payload for sub-command=0x%llx\n",
>> >> +				pkg->hdr.nd_command);
>> >> +			return -EINVAL;
>> >> +		}
>> >> +	}
>> >> +
>> >> +	/* Command looks valid */
>> >
>> > I assume the first command to be implemented also checks the { nd_command,
>> > payload_version, payload length } for correctness?
>> Yes the pdsm service functions do check the payload_version and
>> payload_length. Please see the papr_pdsm_health() that services the
>> PAPR_PDSM_HEALTH pdsm in Patch-5
>> 
>
> cool.
>
>> >
>> >> +	return 0;
>> >> +}
>> >> +
>> >> +static int papr_scm_service_pdsm(struct papr_scm_priv *p,
>> >> +				struct nd_pdsm_cmd_pkg *call_pkg)
>> >> +{
>> >> +	/* unknown subcommands return error in packages */
>> >> +	if (call_pkg->hdr.nd_command <= PAPR_PDSM_MIN ||
>> >> +	    call_pkg->hdr.nd_command >= PAPR_PDSM_MAX) {
>> >> +		dev_dbg(&p->pdev->dev, "Invalid PDSM request 0x%llx\n",
>> >> +			call_pkg->hdr.nd_command);
>> >> +		call_pkg->cmd_status = -EINVAL;
>> >> +		return 0;
>> >> +	}
>> >> +
>> >> +	/* Depending on the DSM command call appropriate service routine */
>> >> +	switch (call_pkg->hdr.nd_command) {
>> >> +	default:
>> >> +		dev_dbg(&p->pdev->dev, "Unsupported PDSM request 0x%llx\n",
>> >> +			call_pkg->hdr.nd_command);
>> >> +		call_pkg->cmd_status = -ENOENT;
>> >> +		return 0;
>> >> +	}
>> >> +}
>> >> +
>> >>  static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>> >>  			  struct nvdimm *nvdimm, unsigned int cmd, void *buf,
>> >>  			  unsigned int buf_len, int *cmd_rc)
>> >>  {
>> >>  	struct nd_cmd_get_config_size *get_size_hdr;
>> >>  	struct papr_scm_priv *p;
>> >> +	struct nd_pdsm_cmd_pkg *call_pkg = NULL;
>> >> +	int rc;
>> >>  
>> >> -	/* Only dimm-specific calls are supported atm */
>> >> -	if (!nvdimm)
>> >> -		return -EINVAL;
>> >> +	/* Use a local variable in case cmd_rc pointer is NULL */
>> >> +	if (cmd_rc == NULL)
>> >> +		cmd_rc = &rc;
>> >
>> > Why is this needed?  AFAICT The caller of papr_scm_ndctl does not specify null
>> > and you did not change it.
>> This pointer is coming from outside the papr_scm code hence need to be
>> defensive here. Also as per[1] cmd_rc is "translation of firmware status"
>> and not every caller would need it hence making this pointer optional.
>> 
>> This is evident in acpi_nfit_blk_get_flags() where the 'nd_desc->ndctl'
>> is called with 'cmd_rc == NULL'.
>> 
>> [1] https://lore.kernel.org/linux-nvdimm/CAPcyv4hE_FG0YZXJVA1G=CBq8b9e0K54jxk5Sq5UKU-dnWT2Kg@mail.gmail.com/
>
> Ah... Ok.  So this is a bug fix which needs to happen regardless of the status
> of this patch...
>
>> 
>> >
>> >> +
>> >> +	*cmd_rc = is_cmd_valid(nvdimm, cmd, buf, buf_len);
>> >> +	if (*cmd_rc) {
>> >> +		pr_debug("Invalid cmd=0x%x. Err=%d\n", cmd, *cmd_rc);
>> >> +		return *cmd_rc;
>> >> +	}
>> >>  
>> >>  	p = nvdimm_provider_data(nvdimm);
>> >>  
>> >> @@ -381,13 +464,19 @@ static int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc,
>> >>  		*cmd_rc = papr_scm_meta_set(p, buf);
>
> ... Because this will break here. even without this new code...  right?
>
> Lets get this fix in as a prelim-patch.
Yes, I have moved the changes proposed in this hunk to a separate prelim
patch in the v10 of the patch series.

>
>> >>  		break;
>> >>  
>> >> +	case ND_CMD_CALL:
>> >> +		call_pkg = nd_to_pdsm_cmd_pkg(buf);
>> >> +		*cmd_rc = papr_scm_service_pdsm(p, call_pkg);
>> >> +		break;
>> >> +
>> >>  	default:
>> >> -		return -EINVAL;
>> >> +		dev_dbg(&p->pdev->dev, "Unknown command = %d\n", cmd);
>> >> +		*cmd_rc = -EINVAL;
>> >
>> > Is this change related?  If there is a bug where there is a caller of
>> > papr_scm_ndctl() with cmd_rc == NULL this should be a separate patch to fix
>> > that issue.
>> This simplifies a bit debugging of errors reported in
>> papr_scm_ndctl() as it ensures that subsequest dev_dbg "Returned with
>> cmd_rc" is always logged.
>> 
>> I think, this is a too small change to be carved out as an independent
>> patch. Also this doesnt change the behaviour of the code except logging
>> some more error info.
>> 
>> However, If you feel too strongly about it I will spin a separate patch
>> in this patch series for this.
[..]
>
> This can go in as part of a 'protect against cmd_rc == NULL'
> preliminary patch.
Yes, have coalesced this change with changes I reffered to in
my previous comment into a single prelim patch.

[..]
>
> I flagged this because at first I could not figure out what this had to do with
> the ND_CMD_CALL...
>
> For reviewers you want to make your patches concise to what you are
> fixing/adding...
Sure. Will be more careful of this in future patches.

>
> Also, based on acpi_nfit_blk_get_flags() using cmd_rc == NULL it looks like we
> have a bug which needs to get fixed regardless of the this patch.  And if that
> bug exists in earlier kernels you will need a separate patch to backport as a
> fix.
>
> So lets get that in first and separate...  :-D
Sure, will send out a separate independent patch fixing the cmd_rc ==
NULL issue in acpi_nfit_blk_get_flags addressing it to stable tree.

>
> Ira
>
>> 
>> >
>> > Ira
>> >
>> >>  	}
>> >>  
>> >>  	dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
>> >>  
>> >> -	return 0;
>> >> +	return *cmd_rc;
>> >>  }
>> >>  
>> >>  static ssize_t flags_show(struct device *dev,
>> >> diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
>> >> index de5d90212409..0e09dc5cec19 100644
>> >> --- a/include/uapi/linux/ndctl.h
>> >> +++ b/include/uapi/linux/ndctl.h
>> >> @@ -244,6 +244,7 @@ struct nd_cmd_pkg {
>> >>  #define NVDIMM_FAMILY_HPE2 2
>> >>  #define NVDIMM_FAMILY_MSFT 3
>> >>  #define NVDIMM_FAMILY_HYPERV 4
>> >> +#define NVDIMM_FAMILY_PAPR 5
>> >>  
>> >>  #define ND_IOCTL_CALL			_IOWR(ND_IOCTL, ND_CMD_CALL,\
>> >>  					struct nd_cmd_pkg)
>> >> -- 
>> >> 2.26.2
>> >> 
>> 
>> -- 
>> Cheers
>> ~ Vaibhav

> -----Original Message-----
> From: Vaibhav Jain <vaibhav@linux.ibm.com>
> Sent: Thursday, June 4, 2020 2:06 AM
> To: Williams, Dan J <dan.j.williams@intel.com>; linuxppc-
> dev@lists.ozlabs.org; linux-nvdimm@lists.01.org; linux-
> kernel@vger.kernel.org
> Cc: Santosh Sivaraj <santosh@fossix.org>; Aneesh Kumar K . V
> <aneesh.kumar@linux.ibm.com>; Steven Rostedt <rostedt@goodmis.org>;
> Oliver O'Halloran <oohall@gmail.com>; Weiny, Ira <ira.weiny@intel.com>
> Subject: RE: [RESEND PATCH v9 4/5] ndctl/papr_scm,uapi: Add support for
> PAPR nvdimm specific methods
> 
> Hi Dan,
> 
> Thanks for review and insights on this. My responses below:
> 
> "Williams, Dan J" <dan.j.williams@intel.com> writes:
> 
> > [ forgive formatting I'm temporarily stuck using Outlook this week...
> > ]
> >
> >> From: Vaibhav Jain <vaibhav@linux.ibm.com>
> > [..]
> >>
> >> Introduce support for PAPR NVDIMM Specific Methods (PDSM) in
> papr_scm
> >> module and add the command family NVDIMM_FAMILY_PAPR to the
> white
> >> list of NVDIMM command sets. Also advertise support for ND_CMD_CALL
> >> for the nvdimm command mask and implement necessary scaffolding in
> >> the module to handle ND_CMD_CALL ioctl and PDSM requests that we
> receive.
> >>
> >> The layout of the PDSM request as we expect from libnvdimm/libndctl
> >> is described in newly introduced uapi header 'papr_pdsm.h' which
> >> defines a new 'struct nd_pdsm_cmd_pkg' header. This header is used to
> >> communicate the PDSM request via member
> 'nd_cmd_pkg.nd_command' and
> >> size of payload that need to be sent/received for servicing the PDSM.
> >>
> >> A new function is_cmd_valid() is implemented that reads the args to
> >> papr_scm_ndctl() and performs sanity tests on them. A new function
> >> papr_scm_service_pdsm() is introduced and is called from
> >> papr_scm_ndctl() in case of a PDSM request is received via
> >> ND_CMD_CALL command from libnvdimm.
> >>
> >> Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
> >> Cc: Dan Williams <dan.j.williams@intel.com>
> >> Cc: Michael Ellerman <mpe@ellerman.id.au>
> >> Cc: Ira Weiny <ira.weiny@intel.com>
> >> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> >> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> >> ---
> >> Changelog:
> >>
> >> Resend:
> >> * Added ack from Aneesh.
> >>
> >> v8..v9:
> >> * Reduced the usage of term SCM replacing it with appropriate
> >>   replacement [ Dan Williams, Aneesh ]
> >> * Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
> >> * s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
> >> * s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
> >> * Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
> >> * Minor update to patch description.
> >>
> >> v7..v8:
> >> * Removed the 'payload_offset' field from 'struct
> >>   nd_pdsm_cmd_pkg'. Instead command payload is always assumed to
> start
> >>   at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
> >> * To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
> >>   'reserved' field of 10-bytes is introduced. [ Aneesh ]
> >> * Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
> >>   [ Ira ]
> >>
> >> Resend:
> >> * None
> >>
> >> v6..v7 :
> >> * Removed the re-definitions of __packed macro from papr_scm_pdsm.h
> >>   [Mpe].
> >> * Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h
> [Mpe].
> >> * Removed macros that were unused in papr_scm.c from
> papr_scm_pdsm.h
> >>   [Mpe].
> >> * Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]
> >>
> >> v5..v6 :
> >> * Changed the usage of the term DSM to PDSM to distinguish it from the
> >>   ACPI term [ Dan Williams ]
> >> * Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various
> >> struct
> >>   to reflect the new terminology.
> >> * Updated the patch description and title to reflect the new terminology.
> >> * Squashed patch to introduce new command family in 'ndctl.h' with
> >>   this patch [ Dan Williams ]
> >> * Updated the papr_scm_pdsm method starting index from 0x10000 to
> 0x0
> >>   [ Dan Williams ]
> >> * Removed redundant license text from the papr_scm_psdm.h file.
> >>   [ Dan Williams ]
> >> * s/envelop/envelope/ at various places [ Dan Williams ]
> >> * Added '__packed' attribute to command package header to gaurd
> >>   against different compiler adding paddings between the fields.
> >>   [ Dan Williams]
> >> * Converted various pr_debug to dev_debug [ Dan Williams ]
> >>
> >> v4..v5 :
> >> * None
> >>
> >> v3..v4 :
> >> * None
> >>
> >> v2..v3 :
> >> * Updated the patch prefix to 'ndctl/uapi' [Aneesh]
> >>
> >> v1..v2 :
> >> * None
> >> ---
> >>  arch/powerpc/include/uapi/asm/papr_pdsm.h | 136
> >> ++++++++++++++++++++++
> arch/powerpc/platforms/pseries/papr_scm.c |
> >> 101 +++++++++++++++-
> >>  include/uapi/linux/ndctl.h                |   1 +
> >>  3 files changed, 232 insertions(+), 6 deletions(-)  create mode
> >> 100644 arch/powerpc/include/uapi/asm/papr_pdsm.h
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/papr_pdsm.h
> >> b/arch/powerpc/include/uapi/asm/papr_pdsm.h
> >> new file mode 100644
> >> index 000000000000..6407fefcc007
> >> --- /dev/null
> >> +++ b/arch/powerpc/include/uapi/asm/papr_pdsm.h
> >> @@ -0,0 +1,136 @@
> >> +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
> >> +/*
> >> + * PAPR nvDimm Specific Methods (PDSM) and structs for libndctl
> >> + *
> >> + * (C) Copyright IBM 2020
> >> + *
> >> + * Author: Vaibhav Jain <vaibhav at linux.ibm.com>  */
> >> +
> >> +#ifndef _UAPI_ASM_POWERPC_PAPR_PDSM_H_ #define
> >> +_UAPI_ASM_POWERPC_PAPR_PDSM_H_
> >> +
> >> +#include <linux/types.h>
> >> +
> >> +/*
> >> + * PDSM Envelope:
> >> + *
> >> + * The ioctl ND_CMD_CALL transfers data between user-space and
> >> +kernel via
> >> + * envelope which consists of a header and user-defined payload
> sections.
> >> + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects
> >> +a
> >> + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload'
> field.
> >> + * There is reserved field that can used to introduce new fields to
> >> +the
> >> + * structure in future. It also tries to ensure that
> >> 'nd_pdsm_cmd_pkg.payload'
> >> + * lies at a 8-byte boundary.
> >> + *
> >> + *  +-------------+---------------------+---------------------------+
> >> + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
> >> + *  +-------------+---------------------+---------------------------+
> >> + *  |               nd_pdsm_cmd_pkg     |                           |
> >> + *  |-------------+                     |                           |
> >> + *  |  nd_cmd_pkg |                     |                           |
> >> + *  +-------------+---------------------+---------------------------+
> >> + *  | nd_family   |                     |                           |
> >> + *  | nd_size_out | cmd_status          |                           |
> >> + *  | nd_size_in  | payload_version     |     payload               |
> >> + *  | nd_command  | reserved            |                           |
> >> + *  | nd_fw_size  |                     |                           |
> >> + *
> >> + +-------------+---------------------+---------------------------+
> >> + *
> >> + * PDSM Header:
> >> + *
> >> + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
> >> + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to
> >> member
> >> + * 'nd_cmd_pkg.nd_command'. Apart from size information of the
> >> envelope
> >> +which is
> >> + * contained in 'struct nd_cmd_pkg', the header also has members
> >> +following
> >> + * members:
> >> + *
> >> + * 'cmd_status'		: (Out) Errors if any encountered while
> >> servicing PDSM.
> >> + * 'payload_version'	: (In/Out) Version number associated with
> the
> >> payload.
> >> + * 'reserved'		: Not used and reserved for future.
> >> + *
> >> + * PDSM Payload:
> >> + *
> >> + * The layout of the PDSM Payload is defined by various structs
> >> +shared between
> >> + * papr_scm and libndctl so that contents of payload can be
> >> +interpreted. During
> >> + * servicing of a PDSM the papr_scm module will read input args from
> >> +the payload
> >> + * field by casting its contents to an appropriate struct pointer
> >> +based on the
> >> + * PDSM command. Similarly the output of servicing the PDSM command
> >> +will be
> >> + * copied to the payload field using the same struct.
> >> + *
> >> + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope
> >> +size, which
> >> + * leaves around 176 bytes for the envelope payload (ignoring any
> >> +padding that
> >> + * the compiler may silently introduce).
> >> + *
> >> + * Payload Version:
> >> + *
> >> + * A 'payload_version' field is present in PDSM header that
> >> +indicates a specific
> >> + * version of the structure present in PDSM Payload for a given PDSM
> >> command.
> >> + * This provides backward compatibility in case the PDSM Payload
> >> +structure
> >> + * evolves and different structures are supported by 'papr_scm' and
> >> 'libndctl'.
> >> + *
> >> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send
> >> +the version
> >> + * of the payload struct it supports via 'payload_version' field.
> >> +The
> >> 'papr_scm'
> >> + * module when servicing the PDSM envelope checks the
> 'payload_version'
> >> +and then
> >> + * uses 'payload struct version' == MIN('payload_version field',
> >> + * 'max payload-struct-version supported by papr_scm') to service
> >> +the
> >> PDSM.
> >> + * After servicing the PDSM, 'papr_scm' put the negotiated version
> >> +of payload
> >> + * struct in returned 'payload_version' field.
> >> + *
> >> + * Libndctl on receiving the envelope back from papr_scm again
> >> +checks the
> >> + * 'payload_version' field and based on it use the appropriate
> >> +version dsm
> >> + * struct to parse the results.
> >> + *
> >> + * Backward Compatibility:
> >> + *
> >> + * Above scheme of exchanging different versioned PDSM struct
> >> +between libndctl
> >> + * and papr_scm should provide backward compatibility until
> >> +following two
> >> + * assumptions/conditions when defining new PDSM structs hold:
> >> + *
> >> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
> >> + *
> >> + * 1. T(X) is a proper subset of T(Y) if Y > X.
> >> + *    i.e Each new version of PDSM struct should retain existing struct
> >> + *    attributes from previous version
> >> + *
> >> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
> >> + *    it should also support T(1), T(2)...T(X - 1).
> >> + *    i.e When adding support for new version of a PDSM struct, libndctl
> >> + *    and papr_scm should retain support of the existing PDSM struct
> >> + *    version they support.
> >> + */
> >> +
> >> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from
> >> libnvdimm
> >> +*/ struct nd_pdsm_cmd_pkg {
> >> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-
> >> cmd */
> >> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
> >> +	__u16 reserved[5];	/* Ignored and to be used in future */
> >> +	__u16 payload_version;	/* In/Out: version of the payload */
> >> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
> >> +} __packed;
> >> +
> >> +/*
> >> + * Methods to be embedded in ND_CMD_CALL request. These are sent
> to
> >> the
> >> +kernel
> >> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
> >> +*/ enum papr_pdsm {
> >> +	PAPR_PDSM_MIN = 0x0,
> >> +	PAPR_PDSM_MAX,
> >> +};
> >> +
> >> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */ static
> >> +inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct
> nd_cmd_pkg
> >> *cmd) {
> >> +	return (struct nd_pdsm_cmd_pkg *) cmd; }
> >> +
> >> +/* Return the payload pointer for a given pcmd */ static inline void
> >> +*pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd) {
> >> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
> >> +		return NULL;
> >> +	else
> >> +		return (void *)(pcmd->payload);
> >> +}
> >> +
> >> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
> >> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c
> >> b/arch/powerpc/platforms/pseries/papr_scm.c
> >> index 149431594839..5e2237e7ec08 100644
> >> --- a/arch/powerpc/platforms/pseries/papr_scm.c
> >> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> >> @@ -15,13 +15,15 @@
> >>  #include <linux/seq_buf.h>
> >>
> >>  #include <asm/plpar_wrappers.h>
> >> +#include <asm/papr_pdsm.h>
> >>
> >>  #define BIND_ANY_ADDR (~0ul)
> >>
> >>  #define PAPR_SCM_DIMM_CMD_MASK \
> >>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
> >>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
> >> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
> >> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
> >> +	 (1ul << ND_CMD_CALL))
> >>
> >>  /* DIMM health bitmap bitmap indicators */
> >>  /* SCM device is unable to persist memory contents */ @@ -350,16
> >> +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
> >>  	return 0;
> >>  }
> >>
> >> +/*
> >> + * Validate the inputs args to dimm-control function and return '0' if valid.
> >> + * This also does initial sanity validation to ND_CMD_CALL
> >> +sub-command
> >> packages.
> >> + */
> >> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd,
> >> +void
> >> *buf,
> >> +		       unsigned int buf_len)
> >> +{
> >> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
> >> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
> >> +	struct papr_scm_priv *p;
> >> +
> >> +	/* Only dimm-specific calls are supported atm */
> >> +	if (!nvdimm)
> >> +		return -EINVAL;
> >> +
> >> +	/* get the provider date from struct nvdimm */
> >> +	p = nvdimm_provider_data(nvdimm);
> >> +
> >> +	if (!test_bit(cmd, &cmd_mask)) {
> >> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
> >> +		return -EINVAL;
> >> +	} else if (cmd == ND_CMD_CALL) {
> >> +
> >> +		/* Verify the envelope package */
> >> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
> >> +				buf_len);
> >> +			return -EINVAL;
> >> +		}
> >> +
> >> +		/* Verify that the PDSM family is valid */
> >> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg
> >> family=0x%llx\n",
> >> +				pkg->hdr.nd_family);
> >> +			return -EINVAL;
> >> +
> >> +		}
> >> +
> >> +		/* We except a payload with all PDSM commands */
> >> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
> >> +			dev_dbg(&p->pdev->dev,
> >> +				"Empty payload for sub-command=0x%llx\n",
> >> +				pkg->hdr.nd_command);
> >> +			return -EINVAL;
> >> +		}
> >> +	}
> >> +
> >> +	/* Command looks valid */
> >
> <snip>
> > So this is where I would expect the kernel to validate the command vs
> > a known list of supported commands / payloads. One of the goals of
> > requiring public documentation of any commands that libnvdimm might
> > support for the ioctl path is to give the kernel the ability to gate
> > future enabling on consideration of a common kernel front-end
> > interface. I believe this would also address questions about the
> > versioning scheme because userspace would be actively prevented from
> > sending command payloads that were not first explicitly enabled in the
> > kernel. This interface as it stands in this patch set seems to be a
> > very thin / "anything goes" passthrough with no consideration for that
> > policy.
> >
> > As an example of the utility of this policy, consider the recent
> > support for nvdimm security commands that allow a passphrase to be set
> > and issue commands like "unlock" and "secure erase". The kernel
> > actively prevents those commands from being sent from userspace. See
> > acpi_nfit_clear_to_send() and nd_cmd_clear_to_send(). The reasoning is
> > that it enforces the kernel's nvdimm security model that uses
> > encrypted/trusted keys to protect key material (clear text keys
> > only-ever exist in kernel-space). Yes, that restriction is painful for
> > people that don't want the kernel's security model and just want the
> > simplicity of passing clear-text keys around, but it's necessary for
> > the kernel to have any chance to provide a common abstraction across
> > vendors. The pain of negotiating every single command with what the
> > kernel will support is useful for the long term health of the kernel.
> > It forces ongoing conversations across vendors to consolidate
> > interfaces and reuse kernel best practices like encrypted/trusted
> > keys. Code acceptance is the only real gate the kernel has to enforce
> > cooperation across vendors.
> >
> > The expectation is that the kernel does not allow any command to pass
> > that is not explicitly listed in a bitmap of known commands. I would
> > expect that if you changed the payload of an existing command that
> > would likely require a new entry in this bitmap. The goal is to give
> > the kernel a chance to constrain the passthrough interface to afford a
> > chance to have a discussion of what might done in a common
> > implementation. Another example is the label-area read-write commands.
> > The kernel needs explicit control to ensure that it owns the label
> > area and that userspace is not able to corrupt it (write it behind the
> > kernel's back).
> >
> > Now that said, I have battle scars with some OEMs that just want a
> > generic passthrough interface so they never need to work with the
> > kernel community again and can just write their custom validation
> > tooling and be done. I've mostly been successful in that fight outside
> > of the gaping hole of ND_CMD_VENDOR. That's the path that ipmctl has
> > used to issue commands that have not made it into the public
> > specification on docs.pmem.io. My warning shot for that is the
> > "disable_vendor_specific" module option that administrators can set to
> > only allow commands that the kernel explicitly knows the effects of to
> > be issued. The result is only tooling / enabling that submits to this
> > auditing regime is guaranteed to work everywhere.
> 
> Agree with points made above. With this patchset we arent really trying to
> push an ioctl passthrough to exchange arbitary data with papr-scm module.
> Nor do we want to bypass the kernel community for any future
> enhancements on this interface. We made some design choices based on
> our understanding of certain restriction we saw in ndctl/libndctl. Specifically
> wanted to avoid issuing two CMD_CALL ioctl roundtrips.
> 
> That being said I had an extended discussion with Aneesh rethinking the
> 'version' field and we both agreed *to remove this field* from the proposed
> 'struct nd_pdsm_cmd_pkg'. This should resolve the contentions around this
> Patch-4 in this patchset. Since the 'version' field isnt extensively used right
> now the impact on the patchset would be small.
> 
> >
> > So, that long explanation out of the way, what does that mean for this
> > patch set? I'd like to understand if you still see a need for a
> > versioning scheme if the implementation is required to explicitly list
> > all the commands it supports? I.e. that the kernel need not worry
> > about userspace sending future unknown payloads because unknown
> > payloads are blocked. Also if your interface has anything similar to a
> > "vendor specific" passthrough I would like to require that go through
> > the ND_CMD_VENDOR ioctl, so that the kernel still has a common check
> > point to prevent vendor specific "I don't want to talk to the kernel
> > community" shenanigans, but even better if ND_CMD_VENDOR is
> something
> > the kernel can eventually jettison because nobody is using it.
> 
> As I mentioned above this isn't a 'vendor specific passthrough'
> machenism. The 'version' field was proposed to avoid two CMD_CALL ioctl
> roundtrip to fetch and report extended nvdimm health data like 'life-
> remaining' which isnt always available for papr-scm.

Oh, why not define a maximal health payload with all the attributes you know about today, leave some room for future expansion, and then report a validity flag for each attribute? This is how the "intel" smart-health payload works. If they ever needed to extend the payload they would increase the size and add more validity flags. Old userspace never groks the new fields, new userspace knows to ask for and parse the larger payload.

See the flags field in 'struct nd_intel_smart' (in ndctl) and the translation of those flags to ndctl generic attribute flags intel_cmd_smart_get_flags().

In general I'd like ndctl to understand the superset of all health attributes across all vendors. For the truly vendor specific ones it would mean that the health flags with a specific "papr_scm" back-end just would never be set on an "intel" device. I.e. look at the "hpe" and "msft" health backends. They only set a subset of the valid flags that could be reported.

> However we just realized instead of relying on 'version' field we can
> advertise support for these extended attributes via nvdimm-flags from sysfs.
> Looking at the nvdimm-flags libndctl can use an appropriate pdsm command
> and struct to fetch the dimm health information from papr_scm via
> CMD_CALL.
> 
> But thats something we plan to do in future and not with the current
> patchset which only reports fixed set of nvdimm health attributes.
> 
> >
> > I feel like this is a conversation that will take a few days to
> > resolve, which does not leave time to push this for v5.8. That said, I
> > do think the health flags patches at the beginning of this series are
> > low risk and uncontentious. How about I merge those for v5.8 and
> > circle back to get this ioctl path queued early in v5.8-rc? Apologies
> > for the late feedback on this relative to v5.8.
> >
> Thanks for this consideration. Agree to the proposal. However changes to
> patchset with removal of 'version' field is fairly small hence can quickly push
> an updated patch series cumulating rest of the review comments from Ira.
> 
> Does that sounds reasonable ?

"Williams, Dan J" <dan.j.williams@intel.com> writes:

>> -----Original Message-----
>> From: Vaibhav Jain <vaibhav@linux.ibm.com>
>> Sent: Thursday, June 4, 2020 2:06 AM
>> To: Williams, Dan J <dan.j.williams@intel.com>; linuxppc-
>> dev@lists.ozlabs.org; linux-nvdimm@lists.01.org; linux-
>> kernel@vger.kernel.org
>> Cc: Santosh Sivaraj <santosh@fossix.org>; Aneesh Kumar K . V
>> <aneesh.kumar@linux.ibm.com>; Steven Rostedt <rostedt@goodmis.org>;
>> Oliver O'Halloran <oohall@gmail.com>; Weiny, Ira <ira.weiny@intel.com>
>> Subject: RE: [RESEND PATCH v9 4/5] ndctl/papr_scm,uapi: Add support for
>> PAPR nvdimm specific methods
>> 
>> Hi Dan,
>> 
>> Thanks for review and insights on this. My responses below:
>> 
>> "Williams, Dan J" <dan.j.williams@intel.com> writes:
>> 
>> > [ forgive formatting I'm temporarily stuck using Outlook this week...
>> > ]
>> >
>> >> From: Vaibhav Jain <vaibhav@linux.ibm.com>
>> > [..]
>> >>
>> >> Introduce support for PAPR NVDIMM Specific Methods (PDSM) in
>> papr_scm
>> >> module and add the command family NVDIMM_FAMILY_PAPR to the
>> white
>> >> list of NVDIMM command sets. Also advertise support for ND_CMD_CALL
>> >> for the nvdimm command mask and implement necessary scaffolding in
>> >> the module to handle ND_CMD_CALL ioctl and PDSM requests that we
>> receive.
>> >>
>> >> The layout of the PDSM request as we expect from libnvdimm/libndctl
>> >> is described in newly introduced uapi header 'papr_pdsm.h' which
>> >> defines a new 'struct nd_pdsm_cmd_pkg' header. This header is used to
>> >> communicate the PDSM request via member
>> 'nd_cmd_pkg.nd_command' and
>> >> size of payload that need to be sent/received for servicing the PDSM.
>> >>
>> >> A new function is_cmd_valid() is implemented that reads the args to
>> >> papr_scm_ndctl() and performs sanity tests on them. A new function
>> >> papr_scm_service_pdsm() is introduced and is called from
>> >> papr_scm_ndctl() in case of a PDSM request is received via
>> >> ND_CMD_CALL command from libnvdimm.
>> >>
>> >> Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
>> >> Cc: Dan Williams <dan.j.williams@intel.com>
>> >> Cc: Michael Ellerman <mpe@ellerman.id.au>
>> >> Cc: Ira Weiny <ira.weiny@intel.com>
>> >> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
>> >> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
>> >> ---
>> >> Changelog:
>> >>
>> >> Resend:
>> >> * Added ack from Aneesh.
>> >>
>> >> v8..v9:
>> >> * Reduced the usage of term SCM replacing it with appropriate
>> >>   replacement [ Dan Williams, Aneesh ]
>> >> * Renamed 'papr_scm_pdsm.h' to 'papr_pdsm.h'
>> >> * s/PAPR_SCM_PDSM_*/PAPR_PDSM_*/g
>> >> * s/NVDIMM_FAMILY_PAPR_SCM/NVDIMM_FAMILY_PAPR/g
>> >> * Minor updates to 'papr_psdm.h' to replace usage of term 'SCM'.
>> >> * Minor update to patch description.
>> >>
>> >> v7..v8:
>> >> * Removed the 'payload_offset' field from 'struct
>> >>   nd_pdsm_cmd_pkg'. Instead command payload is always assumed to
>> start
>> >>   at 'nd_pdsm_cmd_pkg.payload'. [ Aneesh ]
>> >> * To enable introducing new fields to 'struct nd_pdsm_cmd_pkg',
>> >>   'reserved' field of 10-bytes is introduced. [ Aneesh ]
>> >> * Fixed a typo in "Backward Compatibility" section of papr_scm_pdsm.h
>> >>   [ Ira ]
>> >>
>> >> Resend:
>> >> * None
>> >>
>> >> v6..v7 :
>> >> * Removed the re-definitions of __packed macro from papr_scm_pdsm.h
>> >>   [Mpe].
>> >> * Removed the usage of __KERNEL__ macros in papr_scm_pdsm.h
>> [Mpe].
>> >> * Removed macros that were unused in papr_scm.c from
>> papr_scm_pdsm.h
>> >>   [Mpe].
>> >> * Made functions defined in papr_scm_pdsm.h as static inline. [Mpe]
>> >>
>> >> v5..v6 :
>> >> * Changed the usage of the term DSM to PDSM to distinguish it from the
>> >>   ACPI term [ Dan Williams ]
>> >> * Renamed papr_scm_dsm.h to papr_scm_pdsm.h and updated various
>> >> struct
>> >>   to reflect the new terminology.
>> >> * Updated the patch description and title to reflect the new terminology.
>> >> * Squashed patch to introduce new command family in 'ndctl.h' with
>> >>   this patch [ Dan Williams ]
>> >> * Updated the papr_scm_pdsm method starting index from 0x10000 to
>> 0x0
>> >>   [ Dan Williams ]
>> >> * Removed redundant license text from the papr_scm_psdm.h file.
>> >>   [ Dan Williams ]
>> >> * s/envelop/envelope/ at various places [ Dan Williams ]
>> >> * Added '__packed' attribute to command package header to gaurd
>> >>   against different compiler adding paddings between the fields.
>> >>   [ Dan Williams]
>> >> * Converted various pr_debug to dev_debug [ Dan Williams ]
>> >>
>> >> v4..v5 :
>> >> * None
>> >>
>> >> v3..v4 :
>> >> * None
>> >>
>> >> v2..v3 :
>> >> * Updated the patch prefix to 'ndctl/uapi' [Aneesh]
>> >>
>> >> v1..v2 :
>> >> * None
>> >> ---
>> >>  arch/powerpc/include/uapi/asm/papr_pdsm.h | 136
>> >> ++++++++++++++++++++++
>> arch/powerpc/platforms/pseries/papr_scm.c |
>> >> 101 +++++++++++++++-
>> >>  include/uapi/linux/ndctl.h                |   1 +
>> >>  3 files changed, 232 insertions(+), 6 deletions(-)  create mode
>> >> 100644 arch/powerpc/include/uapi/asm/papr_pdsm.h
>> >>
>> >> diff --git a/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> >> b/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> >> new file mode 100644
>> >> index 000000000000..6407fefcc007
>> >> --- /dev/null
>> >> +++ b/arch/powerpc/include/uapi/asm/papr_pdsm.h
>> >> @@ -0,0 +1,136 @@
>> >> +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
>> >> +/*
>> >> + * PAPR nvDimm Specific Methods (PDSM) and structs for libndctl
>> >> + *
>> >> + * (C) Copyright IBM 2020
>> >> + *
>> >> + * Author: Vaibhav Jain <vaibhav at linux.ibm.com>  */
>> >> +
>> >> +#ifndef _UAPI_ASM_POWERPC_PAPR_PDSM_H_ #define
>> >> +_UAPI_ASM_POWERPC_PAPR_PDSM_H_
>> >> +
>> >> +#include <linux/types.h>
>> >> +
>> >> +/*
>> >> + * PDSM Envelope:
>> >> + *
>> >> + * The ioctl ND_CMD_CALL transfers data between user-space and
>> >> +kernel via
>> >> + * envelope which consists of a header and user-defined payload
>> sections.
>> >> + * The header is described by 'struct nd_pdsm_cmd_pkg' which expects
>> >> +a
>> >> + * payload following it and accessible via 'nd_pdsm_cmd_pkg.payload'
>> field.
>> >> + * There is reserved field that can used to introduce new fields to
>> >> +the
>> >> + * structure in future. It also tries to ensure that
>> >> 'nd_pdsm_cmd_pkg.payload'
>> >> + * lies at a 8-byte boundary.
>> >> + *
>> >> + *  +-------------+---------------------+---------------------------+
>> >> + *  |   64-Bytes  |       16-Bytes      |       Max 176-Bytes       |
>> >> + *  +-------------+---------------------+---------------------------+
>> >> + *  |               nd_pdsm_cmd_pkg     |                           |
>> >> + *  |-------------+                     |                           |
>> >> + *  |  nd_cmd_pkg |                     |                           |
>> >> + *  +-------------+---------------------+---------------------------+
>> >> + *  | nd_family   |                     |                           |
>> >> + *  | nd_size_out | cmd_status          |                           |
>> >> + *  | nd_size_in  | payload_version     |     payload               |
>> >> + *  | nd_command  | reserved            |                           |
>> >> + *  | nd_fw_size  |                     |                           |
>> >> + *
>> >> + +-------------+---------------------+---------------------------+
>> >> + *
>> >> + * PDSM Header:
>> >> + *
>> >> + * The header is defined as 'struct nd_pdsm_cmd_pkg' which embeds a
>> >> + * 'struct nd_cmd_pkg' instance. The PDSM command is assigned to
>> >> member
>> >> + * 'nd_cmd_pkg.nd_command'. Apart from size information of the
>> >> envelope
>> >> +which is
>> >> + * contained in 'struct nd_cmd_pkg', the header also has members
>> >> +following
>> >> + * members:
>> >> + *
>> >> + * 'cmd_status'		: (Out) Errors if any encountered while
>> >> servicing PDSM.
>> >> + * 'payload_version'	: (In/Out) Version number associated with
>> the
>> >> payload.
>> >> + * 'reserved'		: Not used and reserved for future.
>> >> + *
>> >> + * PDSM Payload:
>> >> + *
>> >> + * The layout of the PDSM Payload is defined by various structs
>> >> +shared between
>> >> + * papr_scm and libndctl so that contents of payload can be
>> >> +interpreted. During
>> >> + * servicing of a PDSM the papr_scm module will read input args from
>> >> +the payload
>> >> + * field by casting its contents to an appropriate struct pointer
>> >> +based on the
>> >> + * PDSM command. Similarly the output of servicing the PDSM command
>> >> +will be
>> >> + * copied to the payload field using the same struct.
>> >> + *
>> >> + * 'libnvdimm' enforces a hard limit of 256 bytes on the envelope
>> >> +size, which
>> >> + * leaves around 176 bytes for the envelope payload (ignoring any
>> >> +padding that
>> >> + * the compiler may silently introduce).
>> >> + *
>> >> + * Payload Version:
>> >> + *
>> >> + * A 'payload_version' field is present in PDSM header that
>> >> +indicates a specific
>> >> + * version of the structure present in PDSM Payload for a given PDSM
>> >> command.
>> >> + * This provides backward compatibility in case the PDSM Payload
>> >> +structure
>> >> + * evolves and different structures are supported by 'papr_scm' and
>> >> 'libndctl'.
>> >> + *
>> >> + * When sending a PDSM Payload to 'papr_scm', 'libndctl' should send
>> >> +the version
>> >> + * of the payload struct it supports via 'payload_version' field.
>> >> +The
>> >> 'papr_scm'
>> >> + * module when servicing the PDSM envelope checks the
>> 'payload_version'
>> >> +and then
>> >> + * uses 'payload struct version' == MIN('payload_version field',
>> >> + * 'max payload-struct-version supported by papr_scm') to service
>> >> +the
>> >> PDSM.
>> >> + * After servicing the PDSM, 'papr_scm' put the negotiated version
>> >> +of payload
>> >> + * struct in returned 'payload_version' field.
>> >> + *
>> >> + * Libndctl on receiving the envelope back from papr_scm again
>> >> +checks the
>> >> + * 'payload_version' field and based on it use the appropriate
>> >> +version dsm
>> >> + * struct to parse the results.
>> >> + *
>> >> + * Backward Compatibility:
>> >> + *
>> >> + * Above scheme of exchanging different versioned PDSM struct
>> >> +between libndctl
>> >> + * and papr_scm should provide backward compatibility until
>> >> +following two
>> >> + * assumptions/conditions when defining new PDSM structs hold:
>> >> + *
>> >> + * Let T(X) = { set of attributes in PDSM struct 'T' versioned X }
>> >> + *
>> >> + * 1. T(X) is a proper subset of T(Y) if Y > X.
>> >> + *    i.e Each new version of PDSM struct should retain existing struct
>> >> + *    attributes from previous version
>> >> + *
>> >> + * 2. If an entity (libndctl or papr_scm) supports a PDSM struct T(X) then
>> >> + *    it should also support T(1), T(2)...T(X - 1).
>> >> + *    i.e When adding support for new version of a PDSM struct, libndctl
>> >> + *    and papr_scm should retain support of the existing PDSM struct
>> >> + *    version they support.
>> >> + */
>> >> +
>> >> +/* PDSM-header + payload expected with ND_CMD_CALL ioctl from
>> >> libnvdimm
>> >> +*/ struct nd_pdsm_cmd_pkg {
>> >> +	struct nd_cmd_pkg hdr;	/* Package header containing sub-
>> >> cmd */
>> >> +	__s32 cmd_status;	/* Out: Sub-cmd status returned back */
>> >> +	__u16 reserved[5];	/* Ignored and to be used in future */
>> >> +	__u16 payload_version;	/* In/Out: version of the payload */
>> >> +	__u8 payload[];		/* In/Out: Sub-cmd data buffer */
>> >> +} __packed;
>> >> +
>> >> +/*
>> >> + * Methods to be embedded in ND_CMD_CALL request. These are sent
>> to
>> >> the
>> >> +kernel
>> >> + * via 'nd_pdsm_cmd_pkg.hdr.nd_command' member of the ioctl struct
>> >> +*/ enum papr_pdsm {
>> >> +	PAPR_PDSM_MIN = 0x0,
>> >> +	PAPR_PDSM_MAX,
>> >> +};
>> >> +
>> >> +/* Convert a libnvdimm nd_cmd_pkg to pdsm specific pkg */ static
>> >> +inline struct nd_pdsm_cmd_pkg *nd_to_pdsm_cmd_pkg(struct
>> nd_cmd_pkg
>> >> *cmd) {
>> >> +	return (struct nd_pdsm_cmd_pkg *) cmd; }
>> >> +
>> >> +/* Return the payload pointer for a given pcmd */ static inline void
>> >> +*pdsm_cmd_to_payload(struct nd_pdsm_cmd_pkg *pcmd) {
>> >> +	if (pcmd->hdr.nd_size_in == 0 && pcmd->hdr.nd_size_out == 0)
>> >> +		return NULL;
>> >> +	else
>> >> +		return (void *)(pcmd->payload);
>> >> +}
>> >> +
>> >> +#endif /* _UAPI_ASM_POWERPC_PAPR_PDSM_H_ */
>> >> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c
>> >> b/arch/powerpc/platforms/pseries/papr_scm.c
>> >> index 149431594839..5e2237e7ec08 100644
>> >> --- a/arch/powerpc/platforms/pseries/papr_scm.c
>> >> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
>> >> @@ -15,13 +15,15 @@
>> >>  #include <linux/seq_buf.h>
>> >>
>> >>  #include <asm/plpar_wrappers.h>
>> >> +#include <asm/papr_pdsm.h>
>> >>
>> >>  #define BIND_ANY_ADDR (~0ul)
>> >>
>> >>  #define PAPR_SCM_DIMM_CMD_MASK \
>> >>  	((1ul << ND_CMD_GET_CONFIG_SIZE) | \
>> >>  	 (1ul << ND_CMD_GET_CONFIG_DATA) | \
>> >> -	 (1ul << ND_CMD_SET_CONFIG_DATA))
>> >> +	 (1ul << ND_CMD_SET_CONFIG_DATA) | \
>> >> +	 (1ul << ND_CMD_CALL))
>> >>
>> >>  /* DIMM health bitmap bitmap indicators */
>> >>  /* SCM device is unable to persist memory contents */ @@ -350,16
>> >> +352,97 @@ static int papr_scm_meta_set(struct papr_scm_priv *p,
>> >>  	return 0;
>> >>  }
>> >>
>> >> +/*
>> >> + * Validate the inputs args to dimm-control function and return '0' if valid.
>> >> + * This also does initial sanity validation to ND_CMD_CALL
>> >> +sub-command
>> >> packages.
>> >> + */
>> >> +static int is_cmd_valid(struct nvdimm *nvdimm, unsigned int cmd,
>> >> +void
>> >> *buf,
>> >> +		       unsigned int buf_len)
>> >> +{
>> >> +	unsigned long cmd_mask = PAPR_SCM_DIMM_CMD_MASK;
>> >> +	struct nd_pdsm_cmd_pkg *pkg = nd_to_pdsm_cmd_pkg(buf);
>> >> +	struct papr_scm_priv *p;
>> >> +
>> >> +	/* Only dimm-specific calls are supported atm */
>> >> +	if (!nvdimm)
>> >> +		return -EINVAL;
>> >> +
>> >> +	/* get the provider date from struct nvdimm */
>> >> +	p = nvdimm_provider_data(nvdimm);
>> >> +
>> >> +	if (!test_bit(cmd, &cmd_mask)) {
>> >> +		dev_dbg(&p->pdev->dev, "Unsupported cmd=%u\n", cmd);
>> >> +		return -EINVAL;
>> >> +	} else if (cmd == ND_CMD_CALL) {
>> >> +
>> >> +		/* Verify the envelope package */
>> >> +		if (!buf || buf_len < sizeof(struct nd_pdsm_cmd_pkg)) {
>> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg size=%u\n",
>> >> +				buf_len);
>> >> +			return -EINVAL;
>> >> +		}
>> >> +
>> >> +		/* Verify that the PDSM family is valid */
>> >> +		if (pkg->hdr.nd_family != NVDIMM_FAMILY_PAPR) {
>> >> +			dev_dbg(&p->pdev->dev, "Invalid pkg
>> >> family=0x%llx\n",
>> >> +				pkg->hdr.nd_family);
>> >> +			return -EINVAL;
>> >> +
>> >> +		}
>> >> +
>> >> +		/* We except a payload with all PDSM commands */
>> >> +		if (pdsm_cmd_to_payload(pkg) == NULL) {
>> >> +			dev_dbg(&p->pdev->dev,
>> >> +				"Empty payload for sub-command=0x%llx\n",
>> >> +				pkg->hdr.nd_command);
>> >> +			return -EINVAL;
>> >> +		}
>> >> +	}
>> >> +
>> >> +	/* Command looks valid */
>> >
>> <snip>
>> > So this is where I would expect the kernel to validate the command vs
>> > a known list of supported commands / payloads. One of the goals of
>> > requiring public documentation of any commands that libnvdimm might
>> > support for the ioctl path is to give the kernel the ability to gate
>> > future enabling on consideration of a common kernel front-end
>> > interface. I believe this would also address questions about the
>> > versioning scheme because userspace would be actively prevented from
>> > sending command payloads that were not first explicitly enabled in the
>> > kernel. This interface as it stands in this patch set seems to be a
>> > very thin / "anything goes" passthrough with no consideration for that
>> > policy.
>> >
>> > As an example of the utility of this policy, consider the recent
>> > support for nvdimm security commands that allow a passphrase to be set
>> > and issue commands like "unlock" and "secure erase". The kernel
>> > actively prevents those commands from being sent from userspace. See
>> > acpi_nfit_clear_to_send() and nd_cmd_clear_to_send(). The reasoning is
>> > that it enforces the kernel's nvdimm security model that uses
>> > encrypted/trusted keys to protect key material (clear text keys
>> > only-ever exist in kernel-space). Yes, that restriction is painful for
>> > people that don't want the kernel's security model and just want the
>> > simplicity of passing clear-text keys around, but it's necessary for
>> > the kernel to have any chance to provide a common abstraction across
>> > vendors. The pain of negotiating every single command with what the
>> > kernel will support is useful for the long term health of the kernel.
>> > It forces ongoing conversations across vendors to consolidate
>> > interfaces and reuse kernel best practices like encrypted/trusted
>> > keys. Code acceptance is the only real gate the kernel has to enforce
>> > cooperation across vendors.
>> >
>> > The expectation is that the kernel does not allow any command to pass
>> > that is not explicitly listed in a bitmap of known commands. I would
>> > expect that if you changed the payload of an existing command that
>> > would likely require a new entry in this bitmap. The goal is to give
>> > the kernel a chance to constrain the passthrough interface to afford a
>> > chance to have a discussion of what might done in a common
>> > implementation. Another example is the label-area read-write commands.
>> > The kernel needs explicit control to ensure that it owns the label
>> > area and that userspace is not able to corrupt it (write it behind the
>> > kernel's back).
>> >
>> > Now that said, I have battle scars with some OEMs that just want a
>> > generic passthrough interface so they never need to work with the
>> > kernel community again and can just write their custom validation
>> > tooling and be done. I've mostly been successful in that fight outside
>> > of the gaping hole of ND_CMD_VENDOR. That's the path that ipmctl has
>> > used to issue commands that have not made it into the public
>> > specification on docs.pmem.io. My warning shot for that is the
>> > "disable_vendor_specific" module option that administrators can set to
>> > only allow commands that the kernel explicitly knows the effects of to
>> > be issued. The result is only tooling / enabling that submits to this
>> > auditing regime is guaranteed to work everywhere.
>> 
>> Agree with points made above. With this patchset we arent really trying to
>> push an ioctl passthrough to exchange arbitary data with papr-scm module.
>> Nor do we want to bypass the kernel community for any future
>> enhancements on this interface. We made some design choices based on
>> our understanding of certain restriction we saw in ndctl/libndctl. Specifically
>> wanted to avoid issuing two CMD_CALL ioctl roundtrips.
>> 
>> That being said I had an extended discussion with Aneesh rethinking the
>> 'version' field and we both agreed *to remove this field* from the proposed
>> 'struct nd_pdsm_cmd_pkg'. This should resolve the contentions around this
>> Patch-4 in this patchset. Since the 'version' field isnt extensively used right
>> now the impact on the patchset would be small.
>> 
>> >
>> > So, that long explanation out of the way, what does that mean for this
>> > patch set? I'd like to understand if you still see a need for a
>> > versioning scheme if the implementation is required to explicitly list
>> > all the commands it supports? I.e. that the kernel need not worry
>> > about userspace sending future unknown payloads because unknown
>> > payloads are blocked. Also if your interface has anything similar to a
>> > "vendor specific" passthrough I would like to require that go through
>> > the ND_CMD_VENDOR ioctl, so that the kernel still has a common check
>> > point to prevent vendor specific "I don't want to talk to the kernel
>> > community" shenanigans, but even better if ND_CMD_VENDOR is
>> something
>> > the kernel can eventually jettison because nobody is using it.
>> 
>> As I mentioned above this isn't a 'vendor specific passthrough'
>> machenism. The 'version' field was proposed to avoid two CMD_CALL ioctl
>> roundtrip to fetch and report extended nvdimm health data like 'life-
>> remaining' which isnt always available for papr-scm.
>

> Oh, why not define a maximal health payload with all the attributes
> you know about today, leave some room for future expansion, and then
> report a validity flag for each attribute? This is how the "intel"
> smart-health payload works. If they ever needed to extend the payload
> they would increase the size and add more validity flags. Old
> userspace never groks the new fields, new userspace knows to ask for
> and parse the larger payload.
>
> See the flags field in 'struct nd_intel_smart' (in ndctl) and the
> translation of those flags to ndctl generic attribute flags
> intel_cmd_smart_get_flags().
>
> In general I'd like ndctl to understand the superset of all health
> attributes across all vendors. For the truly vendor specific ones it
> would mean that the health flags with a specific "papr_scm" back-end
> just would never be set on an "intel" device. I.e. look at the "hpe"
> and "msft" health backends. They only set a subset of the valid flags
> that could be reported.

Thanks, this sounds good. Infact papr_scm implementation in ndctl does
advertises support for only a subset of ND_SMART_* flags right now.

Using 'flags' instead of 'version' was indeed discussed during
v7..v9. However re-looking at the 'msft' and 'hpe' implementations the
approach of maximal health payload tagged with a flags field looks more
intuitive and I would prefer implementing this scheme in this patch-set.

The current set health data exchanged with between libndctl and
papr_scm via 'struct nd_papr_pdsm_health' (e.g various health status
bits , nvdimm arming status etc) are guaranteed to be always available
hence associating their availability with a flag wont be much useful as
the flag will be always set.

However as you suggested, extending the 'struct nd_papr_pdsm_health' in
future to accommodate new attributes like 'life-remaining' can be done
via adding them to the end of the struct and setting a flag field to
indicate its presence.

So I have the following proposal:
* Add a new '__u32 extension_flags' field at beginning of 'struct
  nd_papr_pdsm_health'
* Set the size of the struct to 184-bytes which is the maximum possible
  size for a pdsm payload.
* 'papr_scm' kernel driver will currently set 'extension_flag' to 0
  indicating no extension fields.

* Future patch that adds support for 'life-remaining' add the new-field
  at the end of known fields in 'struct nd_papr_pdsm_health'.
* When provided to  papr_scm kernel module, if 'life-remaining' data is
  available its populated and corresponding flag set in
  'extension_flags' field indicating its presence.
* When received by libndctl papr_scm implementation its tests if the
  extension_flags have associated 'life-remaining' flag set and if yes
  then return ND_SMART_USED_VALID flag back from
  ndctl_cmd_smart_get_flags().
  
Implementing first 3 items above in the current patchset should be
fairly trivial.

Does that sounds reasonable ?

Thanks,
~ Vaibhav

>
>> However we just realized instead of relying on 'version' field we can
>> advertise support for these extended attributes via nvdimm-flags from sysfs.
>> Looking at the nvdimm-flags libndctl can use an appropriate pdsm command
>> and struct to fetch the dimm health information from papr_scm via
>> CMD_CALL.
>> 
>> But thats something we plan to do in future and not with the current
>> patchset which only reports fixed set of nvdimm health attributes.
>> 
>> >
>> > I feel like this is a conversation that will take a few days to
>> > resolve, which does not leave time to push this for v5.8. That said, I
>> > do think the health flags patches at the beginning of this series are
>> > low risk and uncontentious. How about I merge those for v5.8 and
>> > circle back to get this ioctl path queued early in v5.8-rc? Apologies
>> > for the late feedback on this relative to v5.8.
>> >
>> Thanks for this consideration. Agree to the proposal. However changes to
>> patchset with removal of 'version' field is fairly small hence can quickly push
>> an updated patch series cumulating rest of the review comments from Ira.
>> 
>> Does that sounds reasonable ?
>

On Fri, Jun 5, 2020 at 8:22 AM Vaibhav Jain <vaibhav@linux.ibm.com> wrote:
[..]
> > Oh, why not define a maximal health payload with all the attributes
> > you know about today, leave some room for future expansion, and then
> > report a validity flag for each attribute? This is how the "intel"
> > smart-health payload works. If they ever needed to extend the payload
> > they would increase the size and add more validity flags. Old
> > userspace never groks the new fields, new userspace knows to ask for
> > and parse the larger payload.
> >
> > See the flags field in 'struct nd_intel_smart' (in ndctl) and the
> > translation of those flags to ndctl generic attribute flags
> > intel_cmd_smart_get_flags().
> >
> > In general I'd like ndctl to understand the superset of all health
> > attributes across all vendors. For the truly vendor specific ones it
> > would mean that the health flags with a specific "papr_scm" back-end
> > just would never be set on an "intel" device. I.e. look at the "hpe"
> > and "msft" health backends. They only set a subset of the valid flags
> > that could be reported.
>
> Thanks, this sounds good. Infact papr_scm implementation in ndctl does
> advertises support for only a subset of ND_SMART_* flags right now.
>
> Using 'flags' instead of 'version' was indeed discussed during
> v7..v9. However re-looking at the 'msft' and 'hpe' implementations the
> approach of maximal health payload tagged with a flags field looks more
> intuitive and I would prefer implementing this scheme in this patch-set.
>
> The current set health data exchanged with between libndctl and
> papr_scm via 'struct nd_papr_pdsm_health' (e.g various health status
> bits , nvdimm arming status etc) are guaranteed to be always available
> hence associating their availability with a flag wont be much useful as
> the flag will be always set.
>
> However as you suggested, extending the 'struct nd_papr_pdsm_health' in
> future to accommodate new attributes like 'life-remaining' can be done
> via adding them to the end of the struct and setting a flag field to
> indicate its presence.
>
> So I have the following proposal:
> * Add a new '__u32 extension_flags' field at beginning of 'struct
>   nd_papr_pdsm_health'
> * Set the size of the struct to 184-bytes which is the maximum possible
>   size for a pdsm payload.
> * 'papr_scm' kernel driver will currently set 'extension_flag' to 0
>   indicating no extension fields.
>
> * Future patch that adds support for 'life-remaining' add the new-field
>   at the end of known fields in 'struct nd_papr_pdsm_health'.
> * When provided to  papr_scm kernel module, if 'life-remaining' data is
>   available its populated and corresponding flag set in
>   'extension_flags' field indicating its presence.
> * When received by libndctl papr_scm implementation its tests if the
>   extension_flags have associated 'life-remaining' flag set and if yes
>   then return ND_SMART_USED_VALID flag back from
>   ndctl_cmd_smart_get_flags().
>
> Implementing first 3 items above in the current patchset should be
> fairly trivial.
>
> Does that sounds reasonable ?

This sounds good to me.