mbox series

[v4,00/37] x-blockdev-create for protocols and qcow2

Message ID 20180307185946.29366-1-kwolf@redhat.com
Headers show
Series x-blockdev-create for protocols and qcow2 | expand

Message

Kevin Wolf March 7, 2018, 6:59 p.m. UTC
This series implements a minimal QMP command that allows to create an
image file on the protocol level or an image format on a given block
node.

Eventually, the interface is going to change to some kind of an async
command (possibly a (non-)block job), but that will require more work on
the job infrastructure first, so let's first QAPIfy image creation in
the block drivers. In this series, I'm going for a synchronous command
that is prefixed with x- for now.

This series converts qcow2 and all protocol drivers that allow an actual
image creation. This means that drivers which only check if the already
existing storage is good enough are not converted (e.g. host_device,
iscsi). The old behaviour was useful because 'qemu-img create' wants to
create both protocol and format layer, but with the separation in QMP,
you can just leave out the protocol layer creation when the device
already exists.

Please note that for some of the protocol drivers (gluster, rbd and
sheepdog) I don't have a test setup ready. For those, I only tested
with a fake server address to check that the option are parsed correctly
up to this point and an appropriate error is returned without crashing.

If you are a maintainer of one of these protocols and you are
interested in keeping image creation working for your protocol, you
probably want to test this series on a real setup and give me some
feedback. If you don't, I'll just merge the patches and hope that they
won't break anything.


v4:
- Rebased on top of a few conflicting series that have hit master
  meanwhile
- qcow2: Renamed qcow2_create2() to qcow2_co_create() while resolving
  the conflict from Paolo's series that renamed it to qcow2_co_create2()
- rbd: Further simplified qemu_rbd_mon_host() [Max]


git-backport-diff compared to v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/37:[----] [--] 'block/qapi: Introduce BlockdevCreateOptions'
002/37:[----] [--] 'block/qapi: Add qcow2 create options to schema'
003/37:[down] 'qcow2: Rename qcow2_co_create2() to qcow2_co_create()'
	-> this one is actually new
004/37:[0012] [FC] 'qcow2: Let qcow2_create() handle protocol layer'
005/37:[down] 'qcow2: Pass BlockdevCreateOptions to qcow2_co_create()'
006/37:[down] 'qcow2: Use BlockdevRef in qcow2_co_create()'
007/37:[down] 'qcow2: Use QCryptoBlockCreateOptions in qcow2_co_create()'
008/37:[down] 'qcow2: Handle full/falloc preallocation in qcow2_co_create()'
	-> just rebase with subject line changes
009/37:[----] [--] 'util: Add qemu_opts_to_qdict_filtered()'
010/37:[----] [--] 'test-qemu-opts: Test qemu_opts_append()'
011/37:[----] [--] 'test-qemu-opts: Test qemu_opts_to_qdict_filtered()'
012/37:[----] [--] 'qdict: Introduce qdict_rename_keys()'
013/37:[0005] [FC] 'qcow2: Use visitor for options in qcow2_create()'
014/37:[----] [--] 'block: Make bdrv_is_whitelisted() public'
015/37:[0011] [FC] 'block: x-blockdev-create QMP command'
016/37:[0006] [FC] 'file-posix: Support .bdrv_co_create'
017/37:[0006] [FC] 'file-win32: Support .bdrv_co_create'
018/37:[0018] [FC] 'gluster: Support .bdrv_co_create'
019/37:[----] [--] 'rbd: Fix use after free in qemu_rbd_set_keypairs() error path'
020/37:[----] [--] 'rbd: Factor out qemu_rbd_connect()'
021/37:[----] [--] 'rbd: Remove non-schema options from runtime_opts'
022/37:[0008] [FC] 'rbd: Pass BlockdevOptionsRbd to qemu_rbd_connect()'
023/37:[0022] [FC] 'rbd: Support .bdrv_co_create'
024/37:[----] [--] 'rbd: Assign s->snap/image_name in qemu_rbd_open()'
025/37:[----] [--] 'rbd: Use qemu_rbd_connect() in qemu_rbd_do_create()'
026/37:[----] [-C] 'nfs: Use QAPI options in nfs_client_open()'
027/37:[0006] [FC] 'nfs: Support .bdrv_co_create'
028/37:[----] [-C] 'sheepdog: QAPIfy "redundancy" create option'
029/37:[0009] [FC] 'sheepdog: Support .bdrv_co_create'
030/37:[0001] [FC] 'ssh: Use QAPI BlockdevOptionsSsh object'
031/37:[----] [--] 'ssh: QAPIfy host-key-check option'
032/37:[----] [-C] 'ssh: Pass BlockdevOptionsSsh to connect_to_ssh()'
033/37:[0035] [FC] 'ssh: Support .bdrv_co_create'
034/37:[----] [--] 'file-posix: Fix no-op bdrv_truncate() with falloc preallocation'
035/37:[----] [--] 'block: Fail bdrv_truncate() with negative size'
036/37:[----] [--] 'qemu-iotests: Test qcow2 over file image creation with QMP'
037/37:[----] [--] 'qemu-iotests: Test ssh image creation over QMP'Key:


Kevin Wolf (37):
  block/qapi: Introduce BlockdevCreateOptions
  block/qapi: Add qcow2 create options to schema
  qcow2: Rename qcow2_co_create2() to qcow2_co_create()
  qcow2: Let qcow2_create() handle protocol layer
  qcow2: Pass BlockdevCreateOptions to qcow2_co_create()
  qcow2: Use BlockdevRef in qcow2_co_create()
  qcow2: Use QCryptoBlockCreateOptions in qcow2_co_create()
  qcow2: Handle full/falloc preallocation in qcow2_co_create()
  util: Add qemu_opts_to_qdict_filtered()
  test-qemu-opts: Test qemu_opts_append()
  test-qemu-opts: Test qemu_opts_to_qdict_filtered()
  qdict: Introduce qdict_rename_keys()
  qcow2: Use visitor for options in qcow2_create()
  block: Make bdrv_is_whitelisted() public
  block: x-blockdev-create QMP command
  file-posix: Support .bdrv_co_create
  file-win32: Support .bdrv_co_create
  gluster: Support .bdrv_co_create
  rbd: Fix use after free in qemu_rbd_set_keypairs() error path
  rbd: Factor out qemu_rbd_connect()
  rbd: Remove non-schema options from runtime_opts
  rbd: Pass BlockdevOptionsRbd to qemu_rbd_connect()
  rbd: Support .bdrv_co_create
  rbd: Assign s->snap/image_name in qemu_rbd_open()
  rbd: Use qemu_rbd_connect() in qemu_rbd_do_create()
  nfs: Use QAPI options in nfs_client_open()
  nfs: Support .bdrv_co_create
  sheepdog: QAPIfy "redundancy" create option
  sheepdog: Support .bdrv_co_create
  ssh: Use QAPI BlockdevOptionsSsh object
  ssh: QAPIfy host-key-check option
  ssh: Pass BlockdevOptionsSsh to connect_to_ssh()
  ssh: Support .bdrv_co_create
  file-posix: Fix no-op bdrv_truncate() with falloc preallocation
  block: Fail bdrv_truncate() with negative size
  qemu-iotests: Test qcow2 over file image creation with QMP
  qemu-iotests: Test ssh image creation over QMP

 qapi/block-core.json       | 326 ++++++++++++++++++++++++++++++++-
 include/block/block.h      |   2 +
 include/block/block_int.h  |   5 +-
 include/qapi/qmp/qdict.h   |   6 +
 include/qemu/option.h      |   2 +
 block.c                    |  54 +++++-
 block/create.c             |  76 ++++++++
 block/file-posix.c         |  93 +++++++---
 block/file-win32.c         |  47 ++++-
 block/gluster.c            | 135 +++++++++-----
 block/nfs.c                | 238 +++++++++++--------------
 block/qcow2.c              | 383 +++++++++++++++++++++++++--------------
 block/rbd.c                | 397 ++++++++++++++++++++++-------------------
 block/sheepdog.c           | 321 +++++++++++++++++++++++----------
 block/ssh.c                | 290 ++++++++++++++++--------------
 qobject/qdict.c            |  34 ++++
 tests/check-qdict.c        | 129 ++++++++++++++
 tests/test-qemu-opts.c     | 253 ++++++++++++++++++++++++++
 util/qemu-option.c         |  42 ++++-
 block/Makefile.objs        |   2 +-
 tests/qemu-iotests/049.out |   8 +-
 tests/qemu-iotests/112.out |   4 +-
 tests/qemu-iotests/206     | 436 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/206.out | 209 ++++++++++++++++++++++
 tests/qemu-iotests/207     | 261 +++++++++++++++++++++++++++
 tests/qemu-iotests/207.out |  75 ++++++++
 tests/qemu-iotests/group   |   2 +
 27 files changed, 3056 insertions(+), 774 deletions(-)
 create mode 100644 block/create.c
 create mode 100755 tests/qemu-iotests/206
 create mode 100644 tests/qemu-iotests/206.out
 create mode 100755 tests/qemu-iotests/207
 create mode 100644 tests/qemu-iotests/207.out

Comments

no-reply@patchew.org March 7, 2018, 7:38 p.m. UTC | #1
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180307185946.29366-1-kwolf@redhat.com
Subject: [Qemu-devel] [PATCH v4 00/37] x-blockdev-create for protocols and qcow2

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20180307185946.29366-1-kwolf@redhat.com -> patchew/20180307185946.29366-1-kwolf@redhat.com
 t [tag update]            patchew/cover.1520352600.git.berto@igalia.com -> patchew/cover.1520352600.git.berto@igalia.com
Switched to a new branch 'test'
31a3009b18 qemu-iotests: Test ssh image creation over QMP
04d93a3d4f qemu-iotests: Test qcow2 over file image creation with QMP
4d44559176 block: Fail bdrv_truncate() with negative size
199d11005c file-posix: Fix no-op bdrv_truncate() with falloc preallocation
6d9de3491a ssh: Support .bdrv_co_create
38e4ea119c ssh: Pass BlockdevOptionsSsh to connect_to_ssh()
09a94b57cf ssh: QAPIfy host-key-check option
78fe031228 ssh: Use QAPI BlockdevOptionsSsh object
a43d55b0fe sheepdog: Support .bdrv_co_create
5c16d28edd sheepdog: QAPIfy "redundancy" create option
827ffe113c nfs: Support .bdrv_co_create
4c6f72aa3d nfs: Use QAPI options in nfs_client_open()
cb68550d50 rbd: Use qemu_rbd_connect() in qemu_rbd_do_create()
6534001b69 rbd: Assign s->snap/image_name in qemu_rbd_open()
016039e274 rbd: Support .bdrv_co_create
0fae0f1e6b rbd: Pass BlockdevOptionsRbd to qemu_rbd_connect()
0493c80f19 rbd: Remove non-schema options from runtime_opts
b5e3a19199 rbd: Factor out qemu_rbd_connect()
9f5e2db035 rbd: Fix use after free in qemu_rbd_set_keypairs() error path
a6fe11e442 gluster: Support .bdrv_co_create
8dd9caaaa3 file-win32: Support .bdrv_co_create
561c7126e2 file-posix: Support .bdrv_co_create
61c550d7b9 block: x-blockdev-create QMP command
a349d435e7 block: Make bdrv_is_whitelisted() public
da8d4fde95 qcow2: Use visitor for options in qcow2_create()
0c6082fa17 qdict: Introduce qdict_rename_keys()
54130ce09d test-qemu-opts: Test qemu_opts_to_qdict_filtered()
4876bce8c0 test-qemu-opts: Test qemu_opts_append()
e7a13d4c34 util: Add qemu_opts_to_qdict_filtered()
200a661b86 qcow2: Handle full/falloc preallocation in qcow2_co_create()
b5d9f42cf2 qcow2: Use QCryptoBlockCreateOptions in qcow2_co_create()
070c5be70c qcow2: Use BlockdevRef in qcow2_co_create()
b908bbca7a qcow2: Pass BlockdevCreateOptions to qcow2_co_create()
4339d9e11b qcow2: Let qcow2_create() handle protocol layer
d4b04ac240 qcow2: Rename qcow2_co_create2() to qcow2_co_create()
ada0274302 block/qapi: Add qcow2 create options to schema
ab0ff60240 block/qapi: Introduce BlockdevCreateOptions

=== OUTPUT BEGIN ===
Checking PATCH 1/37: block/qapi: Introduce BlockdevCreateOptions...
Checking PATCH 2/37: block/qapi: Add qcow2 create options to schema...
Checking PATCH 3/37: qcow2: Rename qcow2_co_create2() to qcow2_co_create()...
Checking PATCH 4/37: qcow2: Let qcow2_create() handle protocol layer...
Checking PATCH 5/37: qcow2: Pass BlockdevCreateOptions to qcow2_co_create()...
Checking PATCH 6/37: qcow2: Use BlockdevRef in qcow2_co_create()...
Checking PATCH 7/37: qcow2: Use QCryptoBlockCreateOptions in qcow2_co_create()...
Checking PATCH 8/37: qcow2: Handle full/falloc preallocation in qcow2_co_create()...
Checking PATCH 9/37: util: Add qemu_opts_to_qdict_filtered()...
Checking PATCH 10/37: test-qemu-opts: Test qemu_opts_append()...
Checking PATCH 11/37: test-qemu-opts: Test qemu_opts_to_qdict_filtered()...
WARNING: line over 80 characters
#156: FILE: tests/test-qemu-opts.c:1015:
+    g_test_add_func("/qemu-opts/to_qdict/filtered", test_opts_to_qdict_filtered);

WARNING: line over 80 characters
#157: FILE: tests/test-qemu-opts.c:1016:
+    g_test_add_func("/qemu-opts/to_qdict/duplicates", test_opts_to_qdict_duplicates);

total: 0 errors, 2 warnings, 143 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 12/37: qdict: Introduce qdict_rename_keys()...
Checking PATCH 13/37: qcow2: Use visitor for options in qcow2_create()...
Checking PATCH 14/37: block: Make bdrv_is_whitelisted() public...
Checking PATCH 15/37: block: x-blockdev-create QMP command...
Checking PATCH 16/37: file-posix: Support .bdrv_co_create...
Checking PATCH 17/37: file-win32: Support .bdrv_co_create...
Checking PATCH 18/37: gluster: Support .bdrv_co_create...
Checking PATCH 19/37: rbd: Fix use after free in qemu_rbd_set_keypairs() error path...
Checking PATCH 20/37: rbd: Factor out qemu_rbd_connect()...
Checking PATCH 21/37: rbd: Remove non-schema options from runtime_opts...
Checking PATCH 22/37: rbd: Pass BlockdevOptionsRbd to qemu_rbd_connect()...
Checking PATCH 23/37: rbd: Support .bdrv_co_create...
Checking PATCH 24/37: rbd: Assign s->snap/image_name in qemu_rbd_open()...
Checking PATCH 25/37: rbd: Use qemu_rbd_connect() in qemu_rbd_do_create()...
Checking PATCH 26/37: nfs: Use QAPI options in nfs_client_open()...
Checking PATCH 27/37: nfs: Support .bdrv_co_create...
Checking PATCH 28/37: sheepdog: QAPIfy "redundancy" create option...
Checking PATCH 29/37: sheepdog: Support .bdrv_co_create...
Checking PATCH 30/37: ssh: Use QAPI BlockdevOptionsSsh object...
Checking PATCH 31/37: ssh: QAPIfy host-key-check option...
Checking PATCH 32/37: ssh: Pass BlockdevOptionsSsh to connect_to_ssh()...
Checking PATCH 33/37: ssh: Support .bdrv_co_create...
ERROR: spaces required around that '|' (ctx:VxV)
#47: FILE: block/ssh.c:868:
+                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
                                          ^

ERROR: spaces required around that '|' (ctx:VxE)
#47: FILE: block/ssh.c:868:
+                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
                                                            ^

ERROR: spaces required around that '|' (ctx:VxV)
#48: FILE: block/ssh.c:869:
+                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
                                           ^

total: 3 errors, 0 warnings, 145 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 34/37: file-posix: Fix no-op bdrv_truncate() with falloc preallocation...
WARNING: line over 80 characters
#33: FILE: block/file-posix.c:1690:
+            result = -posix_fallocate(fd, current_length, offset - current_length);

total: 0 errors, 1 warnings, 20 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 35/37: block: Fail bdrv_truncate() with negative size...
Checking PATCH 36/37: qemu-iotests: Test qcow2 over file image creation with QMP...
Checking PATCH 37/37: qemu-iotests: Test ssh image creation over QMP...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
Daniel P. Berrangé March 8, 2018, 10:21 a.m. UTC | #2
On Wed, Mar 07, 2018 at 07:59:09PM +0100, Kevin Wolf wrote:
> This series implements a minimal QMP command that allows to create an
> image file on the protocol level or an image format on a given block
> node.
> 
> Eventually, the interface is going to change to some kind of an async
> command (possibly a (non-)block job), but that will require more work on
> the job infrastructure first, so let's first QAPIfy image creation in
> the block drivers. In this series, I'm going for a synchronous command
> that is prefixed with x- for now.
> 
> This series converts qcow2 and all protocol drivers that allow an actual
> image creation. This means that drivers which only check if the already
> existing storage is good enough are not converted (e.g. host_device,
> iscsi). The old behaviour was useful because 'qemu-img create' wants to
> create both protocol and format layer, but with the separation in QMP,
> you can just leave out the protocol layer creation when the device
> already exists.

Are you going to convert the other format drivers in later versions of
this series, or leave it upto individual maintaniers to convert the
rest ? (personally thinking of the luks driver of course)


Regards,
Daniel
Kevin Wolf March 8, 2018, 11:25 a.m. UTC | #3
Am 08.03.2018 um 11:21 hat Daniel P. Berrangé geschrieben:
> On Wed, Mar 07, 2018 at 07:59:09PM +0100, Kevin Wolf wrote:
> > This series implements a minimal QMP command that allows to create an
> > image file on the protocol level or an image format on a given block
> > node.
> > 
> > Eventually, the interface is going to change to some kind of an async
> > command (possibly a (non-)block job), but that will require more work on
> > the job infrastructure first, so let's first QAPIfy image creation in
> > the block drivers. In this series, I'm going for a synchronous command
> > that is prefixed with x- for now.
> > 
> > This series converts qcow2 and all protocol drivers that allow an actual
> > image creation. This means that drivers which only check if the already
> > existing storage is good enough are not converted (e.g. host_device,
> > iscsi). The old behaviour was useful because 'qemu-img create' wants to
> > create both protocol and format layer, but with the separation in QMP,
> > you can just leave out the protocol layer creation when the device
> > already exists.
> 
> Are you going to convert the other format drivers in later versions of
> this series, or leave it upto individual maintaniers to convert the
> rest ? (personally thinking of the luks driver of course)

In a follow-up series, actually, or perhaps one series per format. This
one is already much longer than I hoped it would become. The longer a
series is, the harder it becomes to get it fully reviewed, address
reviews all over the place and somehow deal with merge conflict. Had I
known how many changes some protocols drivers need, I would probably
have split it into two.

luks happens to be the one format driver that I did first (because it's
alphabetically the first one that supports image creation), so if you
want to look at the patches, they are contained here:

    http://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/blockdev-create

The branch probably doesn't build at the moment, but if you cherry-pick
the luks patches, that should work. I'm going to send them once this
series with the basic support is at least in my block branch (maybe
already later today?)

Kevin