Patchwork [2/2] e2image: add -a switch to include all data

login
register
mail settings
Submitter Phillip Susi
Date Feb. 16, 2012, 9:35 p.m.
Message ID <1329428112-8911-2-git-send-email-psusi@ubuntu.com>
Download mbox | patch
Permalink /patch/141704/
State Superseded
Headers show

Comments

Phillip Susi - Feb. 16, 2012, 9:35 p.m.
Normally the raw and QCOW2 images only contain fs metadata.
Add a new switch ( -a ) to include all data.  This makes it
possible to use e2image to clone a whole filesystem.

Signed-off-by: Phillip Susi <psusi@ubuntu.com>
---
 misc/e2image.8.in |   21 ++++++++++++++++++++-
 misc/e2image.c    |   12 ++++++++----
 2 files changed, 28 insertions(+), 5 deletions(-)
Theodore Ts'o - Feb. 16, 2012, 11:17 p.m.
> +You can convert a qcow2 image into a raw image with:
> +.PP
> +.br
> +\	\fBe2image \-r hda1.qcow2 hda1.raw\fR
> +.br
> +.PP
> +This can be useful to write a qcow2 image containing all data to a
> +sparse image file where it can be loop mounted, or to a disk partition

As I recall Lukas disclaimed a guarantee that the code would work on
qcow2 images that weren't generated by e2image.  (In particular, it
definitely doesn't support compressed or encrypted qcow2 images.)

So we need to make sure we add the appropriate disclaimers that this
might not work on qcow2 images generated by tools other than e2image.

If someone would like to work on improving lib/ext2fs/qcow2.c to add
those missing features, and more importantly, add test cases, it would
be great if our qcow2 support could be made more complete.

> +.PP
> +.SH INCLUDING DATA
> +Normally
> +.B e2image
> +only includes fs metadata, not regular file data.  The 
> +.B \-a
> +option can be specified to include all data.  This will
> +give an image that is suitible to use to clone the entire FS or
> +for backup purposes.  Note that this option only works with the
> +raw or QCOW2 formats.

It will only work for raw images.  The QCOW2 format uses an entirely
different code path, since we don't have an QCOW2 io_manager
abstraction.  That was my original hope, but that's not how our qcow2
support was implemented, so it won't work, and we should probably give
a reasonable warning if someone tries to use the -a flag with anything
other than a raw file system image for e2image's input.

						- Ted

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Phillip Susi - Feb. 17, 2012, 12:17 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/16/2012 06:17 PM, Ted Ts'o wrote:
> As I recall Lukas disclaimed a guarantee that the code would work on
> qcow2 images that weren't generated by e2image.  (In particular, it
> definitely doesn't support compressed or encrypted qcow2 images.)
> 
> So we need to make sure we add the appropriate disclaimers that this
> might not work on qcow2 images generated by tools other than e2image.

Would you care to craft such a disclaimer?

> It will only work for raw images.  The QCOW2 format uses an entirely
> different code path, since we don't have an QCOW2 io_manager
> abstraction.  That was my original hope, but that's not how our qcow2
> support was implemented, so it won't work, and we should probably give
> a reasonable warning if someone tries to use the -a flag with anything
> other than a raw file system image for e2image's input.

It seemed to work fine with qcow2 when I tested it.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPPZyzAAoJEJrBOlT6nu75U2UIAMgAF/+pIFQCiXCQGMqt+6rR
4k2P2Q/NfRIIf5lSuNaMCuV88ZuUU002ywEN6z9hPfxXDIagDAzOArtVI2nAhwfz
Z4BF7uJ+7eJDCDQycNDb4JmAQDRs2sBuu3Y0dHKWthyd39KYMR7B0ARwCSeYLczB
eigCiQwTVechP2jacWwNnv53BP2xbw/BYy5IHfQViSKijxGNdheQkYqOTIO5eGGB
NdPx5A6yogLOHyB3K3E9cqEnAjXSlelLH7sRCPAmHmQiJVscAiNZ6JvyP/oV0uek
HlmvP0LyZwzMvy8XlgoybHNRLysw+VEVVuLIrAzZJHiW7aWH3WPtqrPEQp6XoJY=
=cmZg
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Czerner - Feb. 17, 2012, 9:17 a.m.
On Thu, 16 Feb 2012, Phillip Susi wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 02/16/2012 06:17 PM, Ted Ts'o wrote:
> > As I recall Lukas disclaimed a guarantee that the code would work on
> > qcow2 images that weren't generated by e2image.  (In particular, it
> > definitely doesn't support compressed or encrypted qcow2 images.)
> > 
> > So we need to make sure we add the appropriate disclaimers that this
> > might not work on qcow2 images generated by tools other than e2image.
> 
> Would you care to craft such a disclaimer?
> 
> > It will only work for raw images.  The QCOW2 format uses an entirely
> > different code path, since we don't have an QCOW2 io_manager
> > abstraction.  That was my original hope, but that's not how our qcow2
> > support was implemented, so it won't work, and we should probably give
> > a reasonable warning if someone tries to use the -a flag with anything
> > other than a raw file system image for e2image's input.
> 
> It seemed to work fine with qcow2 when I tested it.

So the problem actually is that I have made some assumptions, like for
example that all data are laid out sequentially (because it is how we
are creating the qcow2 images with e2image) and possibly more. It also
does not support encrypted and compressed images, but it will warn you
about that. And most importantly I have never tested other qcow2 images
other than those generated with e2image, simply because that this is not
the intention of this tool (you have qemu to do that). But if you're able
to test it reliably and create a test case so we can be sure that it
actually works, then we would not need such disclaimer I guess.

Note that you'll have to test that qcow2 images generated by e2image
actually works in other tools reliably (qemu) (again this was not the
intention, we just used qemu2 format because it is space efficient and
non-sparse, so te image can be easily transferred), and that e2image
works well with images generated by other tools. I am not saying that it
would not work (I guess it should), I just never tested it.

The good thing about the qcow2 -> raw conversion done by e2image is that
it is *a lot* faster than qemu implementation (at least when I tested
it). I have never tried to find out why, unfortunately I have not even
tried to see what could be fixed in qemu (I really should:)).

Thanks!
-Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Theodore Ts'o - Feb. 17, 2012, 2:50 p.m.
On Fri, Feb 17, 2012 at 10:17:43AM +0100, Lukas Czerner wrote:
> 
> So the problem actually is that I have made some assumptions, like for
> example that all data are laid out sequentially (because it is how we
> are creating the qcow2 images with e2image) and possibly more. 

So if your code doesn't work correctly if the block numbers are
monotonically increasing in the qcow2 image, I would *have* to assume
there would be plenty of qemu-generated images files that don't have
this constraint (i.e., suppose you write blocks 10000, 42, 16, and 24
while running under qemu, presumably that is the order the blocks
would be written and would show up in the qcow2 file).

Or am I misunderstanding what you are saying here; is it that you are
fairly sure the code will break if the data is not laid out
sequentially, or that you've never tested this case?

Regards,

						- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Czerner - Feb. 17, 2012, 3:35 p.m.
On Fri, 17 Feb 2012, Ted Ts'o wrote:

> On Fri, Feb 17, 2012 at 10:17:43AM +0100, Lukas Czerner wrote:
> > 
> > So the problem actually is that I have made some assumptions, like for
> > example that all data are laid out sequentially (because it is how we
> > are creating the qcow2 images with e2image) and possibly more. 
> 
> So if your code doesn't work correctly if the block numbers are
> monotonically increasing in the qcow2 image, I would *have* to assume
> there would be plenty of qemu-generated images files that don't have
> this constraint (i.e., suppose you write blocks 10000, 42, 16, and 24
> while running under qemu, presumably that is the order the blocks
> would be written and would show up in the qcow2 file).
> 
> Or am I misunderstanding what you are saying here; is it that you are
> fairly sure the code will break if the data is not laid out
> sequentially, or that you've never tested this case?

Hi Ted,

as I said I have never tested this. Well, actually I did some very
limited testing, but since this was not (and still is not right?) the
scope of e2image I did not care enough to test that extensively enough
to be sure that it generally works with qcow2 images other than those
generated with e2image. But I think it should work.

Also I suppose that qemu is being a bit smarter than just writing every
extent out so it does not seek madly back and forth when the image is
not laid out sequentially. e2image does not do that.

Thanks!
-Lukas

> 
> Regards,
> 
> 						- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Czerner - Feb. 17, 2012, 3:39 p.m.
On Fri, 17 Feb 2012, Lukas Czerner wrote:

> On Fri, 17 Feb 2012, Ted Ts'o wrote:
> 
> > On Fri, Feb 17, 2012 at 10:17:43AM +0100, Lukas Czerner wrote:
> > > 
> > > So the problem actually is that I have made some assumptions, like for
> > > example that all data are laid out sequentially (because it is how we
> > > are creating the qcow2 images with e2image) and possibly more. 
> > 
> > So if your code doesn't work correctly if the block numbers are
> > monotonically increasing in the qcow2 image, I would *have* to assume
> > there would be plenty of qemu-generated images files that don't have
> > this constraint (i.e., suppose you write blocks 10000, 42, 16, and 24
> > while running under qemu, presumably that is the order the blocks
> > would be written and would show up in the qcow2 file).
> > 
> > Or am I misunderstanding what you are saying here; is it that you are
> > fairly sure the code will break if the data is not laid out
> > sequentially, or that you've never tested this case?
> 
> Hi Ted,
> 
> as I said I have never tested this. Well, actually I did some very
> limited testing, but since this was not (and still is not right?) the
> scope of e2image I did not care enough to test that extensively enough
> to be sure that it generally works with qcow2 images other than those
> generated with e2image. But I think it should work.
> 
> Also I suppose that qemu is being a bit smarter than just writing every
> extent out so it does not seek madly back and forth when the image is
> not laid out sequentially. e2image does not do that.

ehm..I meant, e2image does not do any smart things there, just seek and
write.

> 
> Thanks!
> -Lukas
> 
> > 
> > Regards,
> > 
> > 						- Ted
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
Theodore Ts'o - Feb. 17, 2012, 8:39 p.m.
On Fri, Feb 17, 2012 at 04:35:20PM +0100, Lukas Czerner wrote:
> as I said I have never tested this. Well, actually I did some very
> limited testing, but since this was not (and still is not right?) the
> scope of e2image

I don't think it's within the scope of e2image to be guaranteed way of
converting a general-purpose qcow2 image file from qemu to a raw image
file.  It would be *nice* if this worked, but I don't think that's the
main thrust of e2image's functionality, no.

I'd consider it more within e2fsprogs's scope if e2fsck, dumpe2fs,
debugfs, etc., could operate directly on a qcow2 file (i.e., if the
qcow2 functionality was implemented as an io_manager).  Combined with
Darrick's fuse2fs, or the e2tools package, it would mean that we could
manipulate qcow2 files directly from the host OS.  I think sysadmins
who are using virtualization would find that to be a useful feature.

I just keep hoping that someone will be able to convince their
management that it's useful enough to put in the development effort.  :-)

	   	     	    	      - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/misc/e2image.8.in b/misc/e2image.8.in
index 74d2a0b..b76dbe8 100644
--- a/misc/e2image.8.in
+++ b/misc/e2image.8.in
@@ -8,7 +8,7 @@  e2image \- Save critical ext2/ext3/ext4 filesystem metadata to a file
 .SH SYNOPSIS
 .B e2image
 [
-.B \-rsI
+.B \-rsIQa
 ]
 .I device
 .I image-file
@@ -171,6 +171,25 @@  is regular QCOW2 image and can be processed by tools aware of QCOW2 format
 such as for example
 .BR qemu-img .
 .PP
+You can convert a qcow2 image into a raw image with:
+.PP
+.br
+\	\fBe2image \-r hda1.qcow2 hda1.raw\fR
+.br
+.PP
+This can be useful to write a qcow2 image containing all data to a
+sparse image file where it can be loop mounted, or to a disk partition
+.PP
+.SH INCLUDING DATA
+Normally
+.B e2image
+only includes fs metadata, not regular file data.  The 
+.B \-a
+option can be specified to include all data.  This will
+give an image that is suitible to use to clone the entire FS or
+for backup purposes.  Note that this option only works with the
+raw or QCOW2 formats.
+.PP
 .SH AUTHOR
 .B e2image 
 was written by Theodore Ts'o (tytso@mit.edu).
diff --git a/misc/e2image.c b/misc/e2image.c
index 722737c..b8f10a1 100644
--- a/misc/e2image.c
+++ b/misc/e2image.c
@@ -52,6 +52,7 @@  extern int optind;
 
 const char * program_name = "e2image";
 char * device_name = NULL;
+char all_data;
 
 static void lseek_error_and_exit(int errnum)
 {
@@ -84,7 +85,7 @@  static int get_bits_from_size(size_t size)
 
 static void usage(void)
 {
-	fprintf(stderr, _("Usage: %s [-rsIQ] device image_file\n"),
+	fprintf(stderr, _("Usage: %s [-rsIQa] device image_file\n"),
 		program_name);
 	exit (1);
 }
@@ -309,7 +310,7 @@  static int process_file_block(ext2_filsys fs EXT2FS_ATTR((unused)),
 			      int ref_offset EXT2FS_ATTR((unused)),
 			      void *priv_data EXT2FS_ATTR((unused)))
 {
-	if (blockcnt < 0) {
+	if (blockcnt < 0 || all_data) {
 		ext2fs_mark_block_bitmap2(meta_block_map, *block_nr);
 		meta_blocks_count++;
 	}
@@ -1114,7 +1115,7 @@  static void write_raw_image_file(ext2_filsys fs, int fd, int type, int flags)
 			if ((inode.i_flags & EXT4_EXTENTS_FL) ||
 			    inode.i_block[EXT2_IND_BLOCK] ||
 			    inode.i_block[EXT2_DIND_BLOCK] ||
-			    inode.i_block[EXT2_TIND_BLOCK]) {
+			    inode.i_block[EXT2_TIND_BLOCK] || all_data) {
 				retval = ext2fs_block_iterate3(fs,
 				       ino, BLOCK_FLAG_READ_ONLY, block_buf,
 				       process_file_block, &pb);
@@ -1242,7 +1243,7 @@  int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 	add_error_table(&et_ext2_error_table);
-	while ((c = getopt(argc, argv, "rsIQ")) != EOF)
+	while ((c = getopt(argc, argv, "rsIQa")) != EOF)
 		switch (c) {
 		case 'I':
 			flags |= E2IMAGE_INSTALL_FLAG;
@@ -1260,6 +1261,9 @@  int main (int argc, char ** argv)
 		case 's':
 			flags |= E2IMAGE_SCRAMBLE_FLAG;
 			break;
+		case 'a':
+			all_data = 1;
+			break;
 		default:
 			usage();
 		}