| Message ID | 1329428112-8911-2-git-send-email-psusi@ubuntu.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
> +You can convert a qcow2 image into a raw image with: > +.PP > +.br > +\ \fBe2image \-r hda1.qcow2 hda1.raw\fR > +.br > +.PP > +This can be useful to write a qcow2 image containing all data to a > +sparse image file where it can be loop mounted, or to a disk partition As I recall Lukas disclaimed a guarantee that the code would work on qcow2 images that weren't generated by e2image. (In particular, it definitely doesn't support compressed or encrypted qcow2 images.) So we need to make sure we add the appropriate disclaimers that this might not work on qcow2 images generated by tools other than e2image. If someone would like to work on improving lib/ext2fs/qcow2.c to add those missing features, and more importantly, add test cases, it would be great if our qcow2 support could be made more complete. > +.PP > +.SH INCLUDING DATA > +Normally > +.B e2image > +only includes fs metadata, not regular file data. The > +.B \-a > +option can be specified to include all data. This will > +give an image that is suitible to use to clone the entire FS or > +for backup purposes. Note that this option only works with the > +raw or QCOW2 formats. It will only work for raw images. The QCOW2 format uses an entirely different code path, since we don't have an QCOW2 io_manager abstraction. That was my original hope, but that's not how our qcow2 support was implemented, so it won't work, and we should probably give a reasonable warning if someone tries to use the -a flag with anything other than a raw file system image for e2image's input. - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/16/2012 06:17 PM, Ted Ts'o wrote: > As I recall Lukas disclaimed a guarantee that the code would work on > qcow2 images that weren't generated by e2image. (In particular, it > definitely doesn't support compressed or encrypted qcow2 images.) > > So we need to make sure we add the appropriate disclaimers that this > might not work on qcow2 images generated by tools other than e2image. Would you care to craft such a disclaimer? > It will only work for raw images. The QCOW2 format uses an entirely > different code path, since we don't have an QCOW2 io_manager > abstraction. That was my original hope, but that's not how our qcow2 > support was implemented, so it won't work, and we should probably give > a reasonable warning if someone tries to use the -a flag with anything > other than a raw file system image for e2image's input. It seemed to work fine with qcow2 when I tested it. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJPPZyzAAoJEJrBOlT6nu75U2UIAMgAF/+pIFQCiXCQGMqt+6rR 4k2P2Q/NfRIIf5lSuNaMCuV88ZuUU002ywEN6z9hPfxXDIagDAzOArtVI2nAhwfz Z4BF7uJ+7eJDCDQycNDb4JmAQDRs2sBuu3Y0dHKWthyd39KYMR7B0ARwCSeYLczB eigCiQwTVechP2jacWwNnv53BP2xbw/BYy5IHfQViSKijxGNdheQkYqOTIO5eGGB NdPx5A6yogLOHyB3K3E9cqEnAjXSlelLH7sRCPAmHmQiJVscAiNZ6JvyP/oV0uek HlmvP0LyZwzMvy8XlgoybHNRLysw+VEVVuLIrAzZJHiW7aWH3WPtqrPEQp6XoJY= =cmZg -----END PGP SIGNATURE----- -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 16 Feb 2012, Phillip Susi wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 02/16/2012 06:17 PM, Ted Ts'o wrote: > > As I recall Lukas disclaimed a guarantee that the code would work on > > qcow2 images that weren't generated by e2image. (In particular, it > > definitely doesn't support compressed or encrypted qcow2 images.) > > > > So we need to make sure we add the appropriate disclaimers that this > > might not work on qcow2 images generated by tools other than e2image. > > Would you care to craft such a disclaimer? > > > It will only work for raw images. The QCOW2 format uses an entirely > > different code path, since we don't have an QCOW2 io_manager > > abstraction. That was my original hope, but that's not how our qcow2 > > support was implemented, so it won't work, and we should probably give > > a reasonable warning if someone tries to use the -a flag with anything > > other than a raw file system image for e2image's input. > > It seemed to work fine with qcow2 when I tested it. So the problem actually is that I have made some assumptions, like for example that all data are laid out sequentially (because it is how we are creating the qcow2 images with e2image) and possibly more. It also does not support encrypted and compressed images, but it will warn you about that. And most importantly I have never tested other qcow2 images other than those generated with e2image, simply because that this is not the intention of this tool (you have qemu to do that). But if you're able to test it reliably and create a test case so we can be sure that it actually works, then we would not need such disclaimer I guess. Note that you'll have to test that qcow2 images generated by e2image actually works in other tools reliably (qemu) (again this was not the intention, we just used qemu2 format because it is space efficient and non-sparse, so te image can be easily transferred), and that e2image works well with images generated by other tools. I am not saying that it would not work (I guess it should), I just never tested it. The good thing about the qcow2 -> raw conversion done by e2image is that it is *a lot* faster than qemu implementation (at least when I tested it). I have never tried to find out why, unfortunately I have not even tried to see what could be fixed in qemu (I really should:)). Thanks! -Lukas -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Feb 17, 2012 at 10:17:43AM +0100, Lukas Czerner wrote: > > So the problem actually is that I have made some assumptions, like for > example that all data are laid out sequentially (because it is how we > are creating the qcow2 images with e2image) and possibly more. So if your code doesn't work correctly if the block numbers are monotonically increasing in the qcow2 image, I would *have* to assume there would be plenty of qemu-generated images files that don't have this constraint (i.e., suppose you write blocks 10000, 42, 16, and 24 while running under qemu, presumably that is the order the blocks would be written and would show up in the qcow2 file). Or am I misunderstanding what you are saying here; is it that you are fairly sure the code will break if the data is not laid out sequentially, or that you've never tested this case? Regards, - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 17 Feb 2012, Ted Ts'o wrote: > On Fri, Feb 17, 2012 at 10:17:43AM +0100, Lukas Czerner wrote: > > > > So the problem actually is that I have made some assumptions, like for > > example that all data are laid out sequentially (because it is how we > > are creating the qcow2 images with e2image) and possibly more. > > So if your code doesn't work correctly if the block numbers are > monotonically increasing in the qcow2 image, I would *have* to assume > there would be plenty of qemu-generated images files that don't have > this constraint (i.e., suppose you write blocks 10000, 42, 16, and 24 > while running under qemu, presumably that is the order the blocks > would be written and would show up in the qcow2 file). > > Or am I misunderstanding what you are saying here; is it that you are > fairly sure the code will break if the data is not laid out > sequentially, or that you've never tested this case? Hi Ted, as I said I have never tested this. Well, actually I did some very limited testing, but since this was not (and still is not right?) the scope of e2image I did not care enough to test that extensively enough to be sure that it generally works with qcow2 images other than those generated with e2image. But I think it should work. Also I suppose that qemu is being a bit smarter than just writing every extent out so it does not seek madly back and forth when the image is not laid out sequentially. e2image does not do that. Thanks! -Lukas > > Regards, > > - Ted > -- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 17 Feb 2012, Lukas Czerner wrote: > On Fri, 17 Feb 2012, Ted Ts'o wrote: > > > On Fri, Feb 17, 2012 at 10:17:43AM +0100, Lukas Czerner wrote: > > > > > > So the problem actually is that I have made some assumptions, like for > > > example that all data are laid out sequentially (because it is how we > > > are creating the qcow2 images with e2image) and possibly more. > > > > So if your code doesn't work correctly if the block numbers are > > monotonically increasing in the qcow2 image, I would *have* to assume > > there would be plenty of qemu-generated images files that don't have > > this constraint (i.e., suppose you write blocks 10000, 42, 16, and 24 > > while running under qemu, presumably that is the order the blocks > > would be written and would show up in the qcow2 file). > > > > Or am I misunderstanding what you are saying here; is it that you are > > fairly sure the code will break if the data is not laid out > > sequentially, or that you've never tested this case? > > Hi Ted, > > as I said I have never tested this. Well, actually I did some very > limited testing, but since this was not (and still is not right?) the > scope of e2image I did not care enough to test that extensively enough > to be sure that it generally works with qcow2 images other than those > generated with e2image. But I think it should work. > > Also I suppose that qemu is being a bit smarter than just writing every > extent out so it does not seek madly back and forth when the image is > not laid out sequentially. e2image does not do that. ehm..I meant, e2image does not do any smart things there, just seek and write. > > Thanks! > -Lukas > > > > > Regards, > > > > - Ted > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
On Fri, Feb 17, 2012 at 04:35:20PM +0100, Lukas Czerner wrote: > as I said I have never tested this. Well, actually I did some very > limited testing, but since this was not (and still is not right?) the > scope of e2image I don't think it's within the scope of e2image to be guaranteed way of converting a general-purpose qcow2 image file from qemu to a raw image file. It would be *nice* if this worked, but I don't think that's the main thrust of e2image's functionality, no. I'd consider it more within e2fsprogs's scope if e2fsck, dumpe2fs, debugfs, etc., could operate directly on a qcow2 file (i.e., if the qcow2 functionality was implemented as an io_manager). Combined with Darrick's fuse2fs, or the e2tools package, it would mean that we could manipulate qcow2 files directly from the host OS. I think sysadmins who are using virtualization would find that to be a useful feature. I just keep hoping that someone will be able to convince their management that it's useful enough to put in the development effort. :-) - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/misc/e2image.8.in b/misc/e2image.8.in index 74d2a0b..b76dbe8 100644 --- a/misc/e2image.8.in +++ b/misc/e2image.8.in @@ -8,7 +8,7 @@ e2image \- Save critical ext2/ext3/ext4 filesystem metadata to a file .SH SYNOPSIS .B e2image [ -.B \-rsI +.B \-rsIQa ] .I device .I image-file @@ -171,6 +171,25 @@ is regular QCOW2 image and can be processed by tools aware of QCOW2 format such as for example .BR qemu-img . .PP +You can convert a qcow2 image into a raw image with: +.PP +.br +\ \fBe2image \-r hda1.qcow2 hda1.raw\fR +.br +.PP +This can be useful to write a qcow2 image containing all data to a +sparse image file where it can be loop mounted, or to a disk partition +.PP +.SH INCLUDING DATA +Normally +.B e2image +only includes fs metadata, not regular file data. The +.B \-a +option can be specified to include all data. This will +give an image that is suitible to use to clone the entire FS or +for backup purposes. Note that this option only works with the +raw or QCOW2 formats. +.PP .SH AUTHOR .B e2image was written by Theodore Ts'o (tytso@mit.edu). diff --git a/misc/e2image.c b/misc/e2image.c index 722737c..b8f10a1 100644 --- a/misc/e2image.c +++ b/misc/e2image.c @@ -52,6 +52,7 @@ extern int optind; const char * program_name = "e2image"; char * device_name = NULL; +char all_data; static void lseek_error_and_exit(int errnum) { @@ -84,7 +85,7 @@ static int get_bits_from_size(size_t size) static void usage(void) { - fprintf(stderr, _("Usage: %s [-rsIQ] device image_file\n"), + fprintf(stderr, _("Usage: %s [-rsIQa] device image_file\n"), program_name); exit (1); } @@ -309,7 +310,7 @@ static int process_file_block(ext2_filsys fs EXT2FS_ATTR((unused)), int ref_offset EXT2FS_ATTR((unused)), void *priv_data EXT2FS_ATTR((unused))) { - if (blockcnt < 0) { + if (blockcnt < 0 || all_data) { ext2fs_mark_block_bitmap2(meta_block_map, *block_nr); meta_blocks_count++; } @@ -1114,7 +1115,7 @@ static void write_raw_image_file(ext2_filsys fs, int fd, int type, int flags) if ((inode.i_flags & EXT4_EXTENTS_FL) || inode.i_block[EXT2_IND_BLOCK] || inode.i_block[EXT2_DIND_BLOCK] || - inode.i_block[EXT2_TIND_BLOCK]) { + inode.i_block[EXT2_TIND_BLOCK] || all_data) { retval = ext2fs_block_iterate3(fs, ino, BLOCK_FLAG_READ_ONLY, block_buf, process_file_block, &pb); @@ -1242,7 +1243,7 @@ int main (int argc, char ** argv) if (argc && *argv) program_name = *argv; add_error_table(&et_ext2_error_table); - while ((c = getopt(argc, argv, "rsIQ")) != EOF) + while ((c = getopt(argc, argv, "rsIQa")) != EOF) switch (c) { case 'I': flags |= E2IMAGE_INSTALL_FLAG; @@ -1260,6 +1261,9 @@ int main (int argc, char ** argv) case 's': flags |= E2IMAGE_SCRAMBLE_FLAG; break; + case 'a': + all_data = 1; + break; default: usage(); }
Normally the raw and QCOW2 images only contain fs metadata. Add a new switch ( -a ) to include all data. This makes it possible to use e2image to clone a whole filesystem. Signed-off-by: Phillip Susi <psusi@ubuntu.com> --- misc/e2image.8.in | 21 ++++++++++++++++++++- misc/e2image.c | 12 ++++++++---- 2 files changed, 28 insertions(+), 5 deletions(-)