[v2] doc: Allow NBD_CMD_FLAG_NO_HOLE during NBD_CMD_WRITE

qemu already has an existing server implementation option that will
explicitly search the payload of NBD_CMD_WRITE for large blocks of
zeroes, and punch holes in the underlying file.  For old clients
that don't know how to use the new NBD_CMD_WRITE_ZEROES, this is a
workaround to keep the server's destination file approximately as
sparse as the client's source.  However, for new clients that know
how to explicitly request holes, it is unnecessary overhead; and
can lead to the server punching a hole and risking fragmentation or
future ENOSPC even when the client explicitly wanted to write
zeroes rather than a hole.  So it makes sense to let the new
NBD_CMD_FLAG_NO_HOLE work for WRITE as well as WRITE_ZEROES.

Signed-off-by: Eric Blake <eblake@redhat.com>
---

v2: fix some typos, add a sentence about server MUST support
NBD_CMD_FLAG_NO_HOLE if it advertises NBD_FLAG_SEND_WRITE_ZEROES

 doc/proto.md | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

Hi,

On Monday 04 April 2016 16:15:43 Eric Blake wrote:
> qemu already has an existing server implementation option that will
> explicitly search the payload of NBD_CMD_WRITE for large blocks of
> zeroes, and punch holes in the underlying file.  For old clients
> that don't know how to use the new NBD_CMD_WRITE_ZEROES, this is a
> workaround to keep the server's destination file approximately as
> sparse as the client's source.  However, for new clients that know
> how to explicitly request holes, it is unnecessary overhead; and
> can lead to the server punching a hole and risking fragmentation or
> future ENOSPC even when the client explicitly wanted to write
> zeroes rather than a hole.  So it makes sense to let the new
> NBD_CMD_FLAG_NO_HOLE work for WRITE as well as WRITE_ZEROES.

From the commit message it sounds like this is only for new clients
supporting WRITE_ZEROES because for those we don't want to search
through all the data of normal WRITEs. If you don't need to set this for
each WRITE individually perhaps we could move it to the negotiation
part?

Best Regards,

Markus

> 
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
> 
> v2: fix some typos, add a sentence about server MUST support
> NBD_CMD_FLAG_NO_HOLE if it advertises NBD_FLAG_SEND_WRITE_ZEROES
> 
>  doc/proto.md | 38 ++++++++++++++++++++++++++++++--------
>  1 file changed, 30 insertions(+), 8 deletions(-)
> 
> diff --git a/doc/proto.md b/doc/proto.md
> index 35a3266..bca0525 100644
> --- a/doc/proto.md
> +++ b/doc/proto.md
> @@ -487,7 +487,7 @@ valid may depend on negotiation during the handshake phase.
>    `NBD_CMD_WRITE_ZEROES` commands.  SHOULD be set to 1 if the client requires
>    "Force Unit Access" mode of operation.  MUST NOT be set unless transmission
>    flags included `NBD_FLAG_SEND_FUA`.
> -- bit 1, `NBD_CMD_NO_HOLE`; defined by the experimental `WRITE_ZEROES`
> +- bit 1, `NBD_CMD_FLAG_NO_HOLE`; defined by the experimental `WRITE_ZEROES`
>    extension; see below.
>  - bit 2, `NBD_CMD_FLAG_DF`; defined by the experimental `STRUCTURED_REPLY`
>    extension; see below
> @@ -574,7 +574,7 @@ The following request types exist:
> 
>      After issuing this command, a client MUST NOT make any assumptions
>      about the contents of the export affected by this command, until
> -    overwriting it again with `NBD_CMD_WRITE`.
> +    overwriting it again with `NBD_CMD_WRITE` or `NBD_CMD_WRITE_ZEROES`.
> 
>      A client MUST NOT send a trim request unless `NBD_FLAG_SEND_TRIM`
>      was set in the transmission flags field.
> @@ -737,8 +737,14 @@ by a sparse file. With current NBD command set, the client has to issue
>  through the wire. The server has to write the data onto disk, effectively
>  losing the sparseness.
> 
> -To remedy this, a `WRITE_ZEROES` extension is envisioned. This extension adds
> -one new command and one new command flag.
> +To remedy this, a `WRITE_ZEROES` extension is envisioned. This
> +extension adds one new transmission flag, one new command, and one new
> +command flag; and refines an existing command.
> +
> +* `NBD_FLAG_SEND_WRITE_ZEROES`
> +
> +    The server SHOULD set this transmission flag to 1 if the
> +    `NBD_CMD_WRITE_ZEROES` request is supported.
> 
>  * `NBD_CMD_WRITE_ZEROES`
> 
> @@ -772,12 +778,28 @@ The server SHOULD return `ENOSPC` if it receives a write zeroes request
>  including one or more sectors beyond the size of the device. It SHOULD
>  return `EPERM` if it receives a write zeroes request on a read-only export.
> 
> +* `NBD_CMD_WRITE`
> +
> +    By default, the server MAY search for large contiguous blocks of
> +    all zero content, and use trimming to zero out those portions of
> +    the write, even if it did not advertise `NBD_FLAG_SEND_TRIM`; but
> +    it MUST ensure that any trimmed areas of data read back as zero.
> +    However, the client MAY set the command flag
> +    `NBD_CMD_FLAG_NO_HOLE` to inform the server that the entire
> +    written area MUST be fully provisioned, ensuring that future
> +    writes to the same area will not cause fragmentation or cause
> +    failure due to insufficient space.  Clients SHOULD NOT set this
> +    flag unless the server advertised `NBD_FLAG_SEND_WRITE_ZEROES` in
> +    the transmission flags.
> +
>  The extension adds the following new command flag:
> 
> -- `NBD_CMD_FLAG_NO_HOLE`; valid during `NBD_CMD_WRITE_ZEROES`.
> -  SHOULD be set to 1 if the client wants to ensure that the server does
> -  not create a hole. The client MAY send `NBD_CMD_FLAG_NO_HOLE` even
> -  if `NBD_FLAG_SEND_TRIM` was not set in the transmission flags field.
> +- `NBD_CMD_FLAG_NO_HOLE`; valid during `NBD_CMD_WRITE` and
> +  `NBD_CMD_WRITE_ZEROES`.  SHOULD be set to 1 if the client wants to
> +  ensure that the server does not create a hole. The client MAY send
> +  `NBD_CMD_FLAG_NO_HOLE` even if `NBD_FLAG_SEND_TRIM` was not set in
> +  the transmission flags field.  The server MUST support the use of
> +  this flag if it advertises `NBD_FLAG_SEND_WRITE_ZEROES`.
> 
>  ### `STRUCTURED_REPLY` extension
> 
>

On 04/05/2016 03:38 AM, Markus Pargmann wrote:
> Hi,
> 
> On Monday 04 April 2016 16:15:43 Eric Blake wrote:
>> qemu already has an existing server implementation option that will
>> explicitly search the payload of NBD_CMD_WRITE for large blocks of
>> zeroes, and punch holes in the underlying file.  For old clients
>> that don't know how to use the new NBD_CMD_WRITE_ZEROES, this is a
>> workaround to keep the server's destination file approximately as
>> sparse as the client's source.  However, for new clients that know
>> how to explicitly request holes, it is unnecessary overhead; and
>> can lead to the server punching a hole and risking fragmentation or
>> future ENOSPC even when the client explicitly wanted to write
>> zeroes rather than a hole.  So it makes sense to let the new
>> NBD_CMD_FLAG_NO_HOLE work for WRITE as well as WRITE_ZEROES.
> 
> From the commit message it sounds like this is only for new clients
> supporting WRITE_ZEROES because for those we don't want to search
> through all the data of normal WRITEs. If you don't need to set this for
> each WRITE individually perhaps we could move it to the negotiation
> part?

Interesting idea.  So we'd add a new NBD_OPT_XXX that lets the server
know that "I plan on using WRITE_ZEROS and TRIM as the only places where
I want you to trim, so you can avoid scanning for zeroes in WRITE"; the
server replies with NBD_REP_ACK if it understands the client (in which
case the server _should_ be advertising NBD_FLAG_SEND_WRITE_ZEROES
and/or NBD_FLAG_SEND_TRIM), and with NBD_REP_ERR_UNSUP if it is too old
(the server may still advertise TRIM, but probably should not advertise
WRITE_ZEROES - we are still early enough that we could mandate that any
server that supports WRITE_ZEROES also supports the new NBD_OPT_XXX).
The client then knows that either the server will be efficient with
WRITE and the client uses WRITE_ZEROES and TRIM as desired, or that the
server is old and the client is stuck with WRITE anyways (and whether
the server trims or not is beyond the client's control).

Meanwhile, the new server can unconditionally advertise SEND_TRIM and
SEND_WRITE_ZEROES, whether or not a client uses NBD_OPT_XXX.  If it is
an old client connecting and no NBD_OPT_XXX is sent, chances are the
client is also too old to ever use WRITE_ZEROES, so the server should
feel free to apply its policy on whether to scan for zeroes in WRITE (in
qemu's case, the server policy is set via command line options,
precisely to cater to the scenario where we WANT the server to scan
zeroes to make up for the client being unable to pass sparse regions
efficiently vs. cases where the scanning is deemed too expensive); but
if the client DID send NBD_OPT_XXX, the server SHOULD NOT punch holes
during WRITE, and should not waste time scanning, no matter what the
command line policy permitted.

This also helps the case of clients divided between userspace and
kernel: the way I wrote the proposal, the kernel has to pay attention to
NBD_FLAG_SEND_WRITE_ZEROES, and if present, add NBD_CMD_FLAG_NO_HOLE to
every write.  But with your proposal of option negotiation (done in
userspace), the default of WRITE is now the most efficient on new
servers, and unchanged for old servers, so the kernel doesn't have to do
anything different.

Does the idea of a new NBD_OPT_ make enough sense to write that up
rather than mandating the use of NBD_CMD_FLAG_NO_HOLE with WRITE?

On Tue, Apr 05, 2016 at 10:43:14AM -0600, Eric Blake wrote:
> On 04/05/2016 03:38 AM, Markus Pargmann wrote:
> > Hi,
> > 
> > On Monday 04 April 2016 16:15:43 Eric Blake wrote:
> >> qemu already has an existing server implementation option that will
> >> explicitly search the payload of NBD_CMD_WRITE for large blocks of
> >> zeroes, and punch holes in the underlying file.  For old clients
> >> that don't know how to use the new NBD_CMD_WRITE_ZEROES, this is a
> >> workaround to keep the server's destination file approximately as
> >> sparse as the client's source.  However, for new clients that know
> >> how to explicitly request holes, it is unnecessary overhead; and
> >> can lead to the server punching a hole and risking fragmentation or
> >> future ENOSPC even when the client explicitly wanted to write
> >> zeroes rather than a hole.  So it makes sense to let the new
> >> NBD_CMD_FLAG_NO_HOLE work for WRITE as well as WRITE_ZEROES.
> > 
> > From the commit message it sounds like this is only for new clients
> > supporting WRITE_ZEROES because for those we don't want to search
> > through all the data of normal WRITEs. If you don't need to set this for
> > each WRITE individually perhaps we could move it to the negotiation
> > part?
> 
> Interesting idea.  So we'd add a new NBD_OPT_XXX that lets the server
> know that "I plan on using WRITE_ZEROS and TRIM as the only places where
> I want you to trim, so you can avoid scanning for zeroes in WRITE"; the
> server replies with NBD_REP_ACK if it understands the client (in which
> case the server _should_ be advertising NBD_FLAG_SEND_WRITE_ZEROES
> and/or NBD_FLAG_SEND_TRIM), and with NBD_REP_ERR_UNSUP if it is too old
> (the server may still advertise TRIM, but probably should not advertise
> WRITE_ZEROES - we are still early enough that we could mandate that any
> server that supports WRITE_ZEROES also supports the new NBD_OPT_XXX).

Certainly.

However, I think a server should be allowed to reply to this
NBD_OPT_NO_AUTO_HOLE (or whatever we end up calling it) with
NBD_REP_ERR_POLICY -- i.e., it understands the request, but server-side
configuration forbids it to heed it.

This kind of stuff is *always* a trade-off. Someone low on diskspace
might want to force their server to scan for zeroes, in the
understanding that things might break.

[...]
> Does the idea of a new NBD_OPT_ make enough sense to write that up
> rather than mandating the use of NBD_CMD_FLAG_NO_HOLE with WRITE?

Yeah, it does to me. The client shouldn't have to care much about this
kind of stuff.