mbox series

[v6,00/32] Add subcluster allocation to qcow2

Message ID cover.1590331741.git.berto@igalia.com
Headers show
Series Add subcluster allocation to qcow2 | expand

Message

Alberto Garcia May 24, 2020, 2:51 p.m. UTC 
Hi,

here's the new version of the patches to add subcluster allocation
support to qcow2.

Please refer to the cover letter of the first version for a full
description of the patches:

   https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

The big change here is that the code does not need to iterate over
every subcluster anymore. There's now a new function called
qcow2_get_subcluster_range_type() that checks a group of subclusters
in one go using cto and ctz operations.

Berto

v6:
- Patch 07: Fix indentation issues in the documentation [Eric]
- Patch 14: Use half-open intervals for the subcluster macros.
            New method to detected invalid subclusters.
- Patch 15: New patch
- Patch 20: Use a faster method to check the status of a groups of
            subclusters without having to iterate over each one of
            them.
- Patch 21: Optimize count_contiguous_subclusters() by removing the
            iteration over every subcluster.
            Earlier detection of invalid subclusters.
            Minor documentation updates [Eric]
- Patch 25: Update after the macro changes in patch 14
- Patch 28: Update after the macro changes in patch 14
- Patch 29: Change patch order so this goes before patch 30
- Patch 30: Fix rebase conflicts now that compression_type has been
            merged.
            Disable toggling extended_l2 using amend [Eric]
- Patch 32: Many new tests and refactoring

v5: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg00251.html
v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html
v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Output of git backport-diff against v5:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/32:[----] [--] 'qcow2: Make Qcow2AioTask store the full host offset'
002/32:[----] [--] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()'
003/32:[----] [--] 'qcow2: Add calculate_l2_meta()'
004/32:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()'
005/32:[----] [--] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()'
006/32:[----] [--] 'qcow2: Add get_l2_entry() and set_l2_entry()'
007/32:[0006] [FC] 'qcow2: Document the Extended L2 Entries feature'
008/32:[----] [--] 'qcow2: Add dummy has_subclusters() function'
009/32:[----] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
010/32:[----] [--] 'qcow2: Add offset_to_sc_index()'
011/32:[----] [--] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()'
012/32:[----] [--] 'qcow2: Add l2_entry_size()'
013/32:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()'
014/32:[0039] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()'
015/32:[down] 'qcow2: Add qcow2_get_subcluster_range_type()'
016/32:[----] [--] 'qcow2: Add qcow2_cluster_is_allocated()'
017/32:[----] [--] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()'
018/32:[----] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
019/32:[----] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
020/32:[0111] [FC] 'qcow2: Add subcluster support to calculate_l2_meta()'
021/32:[0091] [FC] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
022/32:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()'
023/32:[----] [--] 'qcow2: Add subcluster support to discard_in_l2_slice()'
024/32:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()'
025/32:[0004] [FC] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
026/32:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster'
027/32:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
028/32:[0004] [FC] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()'
029/32:[0001] [FC] 'qcow2: Add subcluster support to qcow2_measure()'
030/32:[0549] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit'
031/32:[----] [--] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters'
032/32:[0775] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries'

Alberto Garcia (32):
  qcow2: Make Qcow2AioTask store the full host offset
  qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()
  qcow2: Add calculate_l2_meta()
  qcow2: Split cluster_needs_cow() out of count_cow_clusters()
  qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()
  qcow2: Add get_l2_entry() and set_l2_entry()
  qcow2: Document the Extended L2 Entries feature
  qcow2: Add dummy has_subclusters() function
  qcow2: Add subcluster-related fields to BDRVQcow2State
  qcow2: Add offset_to_sc_index()
  qcow2: Add offset_into_subcluster() and size_to_subclusters()
  qcow2: Add l2_entry_size()
  qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()
  qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()
  qcow2: Add qcow2_get_subcluster_range_type()
  qcow2: Add qcow2_cluster_is_allocated()
  qcow2: Add cluster type parameter to qcow2_get_host_offset()
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  qcow2: Add subcluster support to calculate_l2_meta()
  qcow2: Add subcluster support to qcow2_get_host_offset()
  qcow2: Add subcluster support to zero_in_l2_slice()
  qcow2: Add subcluster support to discard_in_l2_slice()
  qcow2: Add subcluster support to check_refcounts_l2()
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
  qcow2: Clear the L2 bitmap when allocating a compressed cluster
  qcow2: Add subcluster support to handle_alloc_space()
  qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
  qcow2: Add subcluster support to qcow2_measure()
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
  qcow2: Assert that expand_zero_clusters_in_l1() does not support
    subclusters
  iotests: Add tests for qcow2 images with extended L2 entries

 docs/interop/qcow2.txt           |  68 ++-
 docs/qcow2-cache.txt             |  19 +-
 qapi/block-core.json             |   7 +
 block/qcow2.h                    | 203 ++++++-
 include/block/block_int.h        |   1 +
 block/qcow2-cluster.c            | 911 +++++++++++++++++++++----------
 block/qcow2-refcount.c           |  38 +-
 block/qcow2.c                    | 291 ++++++----
 block/trace-events               |   2 +-
 tests/qemu-iotests/031.out       |   8 +-
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++--
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061           |   6 +
 tests/qemu-iotests/061.out       |  25 +-
 tests/qemu-iotests/065           |  12 +-
 tests/qemu-iotests/082.out       |  48 +-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/198.out       |   2 +
 tests/qemu-iotests/206.out       |   4 +
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/271           | 705 ++++++++++++++++++++++++
 tests/qemu-iotests/271.out       | 603 ++++++++++++++++++++
 tests/qemu-iotests/274.out       |  49 +-
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/common.filter |   1 +
 tests/qemu-iotests/group         |   1 +
 31 files changed, 2616 insertions(+), 562 deletions(-)
 create mode 100755 tests/qemu-iotests/271
 create mode 100644 tests/qemu-iotests/271.out

Comments

no-reply@patchew.org May 24, 2020, 4:29 p.m. UTC | #1 
Patchew URL: https://patchew.org/QEMU/cover.1590331741.git.berto@igalia.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/qed-table.o
  CC      block/qed-cluster.o
/tmp/qemu-test/src/block/qcow2-cluster.c: In function 'qcow2_get_host_offset':
/tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (type != expected_type) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
     QCow2SubclusterType expected_type, type;
                         ^
/tmp/qemu-test/src/block/qcow2-cluster.c:475:19: error: 'check_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (check_offset) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:447:10: note: 'check_offset' was declared here
     bool check_offset;
          ^
/tmp/qemu-test/src/block/qcow2-cluster.c:476:29: error: 'expected_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
             expected_offset += s->cluster_size;
                             ^
/tmp/qemu-test/src/block/qcow2-cluster.c:448:14: note: 'expected_offset' was declared here
     uint64_t expected_offset;
              ^
cc1: all warnings being treated as errors
make: *** [block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=5d5c1a3ec20c4733b2b49cfa8c25190e', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-ke8h2yeo/src/docker-src.2020-05-24-12.27.22.23564:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=5d5c1a3ec20c4733b2b49cfa8c25190e
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-ke8h2yeo/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m28.041s
user    0m8.588s


The full log is available at
http://patchew.org/logs/cover.1590331741.git.berto@igalia.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org May 24, 2020, 4:34 p.m. UTC | #2 
Patchew URL: https://patchew.org/QEMU/cover.1590331741.git.berto@igalia.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
/tmp/qemu-test/src/block/qcow2-cluster.c:1912:54: error: implicit conversion from enumeration type 'QCow2ClusterType' (aka 'enum QCow2ClusterType') to different enumeration type 'enum qcow2_discard_type' [-Werror,-Wenum-conversion]
        qcow2_free_any_clusters(bs, old_l2_entry, 1, type);
        ~~~~~~~~~~~~~~~~~~~~~~~                      ^~~~
1 error generated.
make: *** [/tmp/qemu-test/src/rules.mak:69: block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=247935bf1140449991b4816970528c03', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-59iulxz0/src/docker-src.2020-05-24-12.30.45.31580:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=247935bf1140449991b4816970528c03
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-59iulxz0/src'
make: *** [docker-run-test-debug@fedora] Error 2

real    3m31.674s
user    0m8.521s


The full log is available at
http://patchew.org/logs/cover.1590331741.git.berto@igalia.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org May 24, 2020, 4:42 p.m. UTC | #3 
Patchew URL: https://patchew.org/QEMU/cover.1590331741.git.berto@igalia.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/snapshot.o
  CC      block/qapi.o
/tmp/qemu-test/src/block/qcow2-cluster.c: In function 'qcow2_get_host_offset':
/tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (type != expected_type) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
     QCow2SubclusterType expected_type, type;
                         ^
/tmp/qemu-test/src/block/qcow2-cluster.c:475:19: error: 'check_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (check_offset) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:447:10: note: 'check_offset' was declared here
     bool check_offset;
          ^
/tmp/qemu-test/src/block/qcow2-cluster.c:476:29: error: 'expected_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
             expected_offset += s->cluster_size;
                             ^
/tmp/qemu-test/src/block/qcow2-cluster.c:448:14: note: 'expected_offset' was declared here
     uint64_t expected_offset;
              ^
cc1: all warnings being treated as errors
make: *** [block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=2b00ed76657046888b159ce8f43ddafe', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-8sa3xlcn/src/docker-src.2020-05-24-12.39.44.4969:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=2b00ed76657046888b159ce8f43ddafe
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-8sa3xlcn/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m27.867s
user    0m4.586s


The full log is available at
http://patchew.org/logs/cover.1590331741.git.berto@igalia.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org May 24, 2020, 4:46 p.m. UTC | #4 
Patchew URL: https://patchew.org/QEMU/cover.1590331741.git.berto@igalia.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/parallels.o
  CC      block/blklogwrites.o
  CC      block/block-backend.o
/tmp/qemu-test/src/block/qcow2-cluster.c:1912:54: error: implicit conversion from enumeration type 'QCow2ClusterType' (aka 'enum QCow2ClusterType') to different enumeration type 'enum qcow2_discard_type' [-Werror,-Wenum-conversion]
        qcow2_free_any_clusters(bs, old_l2_entry, 1, type);
        ~~~~~~~~~~~~~~~~~~~~~~~                      ^~~~
1 error generated.
make: *** [/tmp/qemu-test/src/rules.mak:69: block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=7888e00ead4743f2936bfa5f2bc1cd85', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-a3y14gp6/src/docker-src.2020-05-24-12.43.26.11536:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=7888e00ead4743f2936bfa5f2bc1cd85
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-a3y14gp6/src'
make: *** [docker-run-test-debug@fedora] Error 2

real    3m28.097s
user    0m7.204s


The full log is available at
http://patchew.org/logs/cover.1590331741.git.berto@igalia.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Alberto Garcia May 25, 2020, 4:06 p.m. UTC | #5 
On Sun 24 May 2020 06:34:17 PM CEST, no-reply@patchew.org wrote:
> /tmp/qemu-test/src/block/qcow2-cluster.c:1912:54: error: implicit conversion from enumeration type 'QCow2ClusterType' (aka 'enum QCow2ClusterType') to different enumeration type 'enum qcow2_discard_type' [-Werror,-Wenum-conversion]
>         qcow2_free_any_clusters(bs, old_l2_entry, 1, type);
>         ~~~~~~~~~~~~~~~~~~~~~~~                      ^~~~

This is actually a bug, I'll send a new version with this fixed.

Berto