| Message ID | 1424473645-29161-20-git-send-email-jsnow@redhat.com |
|---|---|
| State | New |
| Headers | show |
On 02/20/2015 04:07 PM, John Snow wrote: > Signed-off-by: John Snow <jsnow@redhat.com> > --- > docs/bitmaps.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 253 insertions(+) > create mode 100644 docs/bitmaps.md > > diff --git a/docs/bitmaps.md b/docs/bitmaps.md > new file mode 100644 > index 0000000..7cda146 > --- /dev/null > +++ b/docs/bitmaps.md > @@ -0,0 +1,253 @@ > +# Dirty Bitmaps No copyright/license? Might be good to add. Also, I'm a fan of reviewing docs before code (does the design make sense in isolation, and then did we implement it) rather than last in the series (we may have faithfully documented our code, but that includes baking in any design flaws that reviewers are now blind to because of reading the code) > +* To create a new bitmap that tracks changes in 32KiB segments: > + > +```json > +{ "execute": "block-dirty-bitmap-add", > + "arguments": { > + "node": "drive0", > + "name": "bitmap0", > + "granularity": "32768" s/"32768"/32768/ (we are using a JSON integer, not string) > + } > +} > +``` > + > +### Deletion > + > +* Can be performed on a disabled bitmap, but not a frozen one. > + > +* Because bitmaps are only unique to the node to which they are attached, > +you must specify the node/drive name here, too. > + > +```json > +{ "execute": "block-dirty-bitmap-remove", > + "arguments": { > + "node": "drive0", > + "name": "bitmap0", > + } No trailing commas in JSON. > +} > +``` > + > +### Enable/Disable: > + > +* Not very useful in current cases, but potentially useful for debugging in the > +future where we'd like to see what information changed only in a specific > +time period: > + > +* To enable (which is, again, the default state after add) > + > +```json > +{ "execute": "block-dirty-bitmap-enable", > + "arguments": { > + "node": "drive0", > + "name": "bitmap0", > + } and again. > +} > +``` > + > +* To disable: > + > +```json > +{ "execute": "block-dirty-bitmap-disable", > + "arguments": { > + "node": "drive0", > + "name": "bitmap0", > + } > +} > +``` and again. Also, maybe swap these two, since a bitmap defaults to enabled (that is, you would logically use disable first, then enable to re-enable, when testing these out, if there is no way to create an already-disabled bitmap). > + > +### Resetting > + > +* Resetting a bitmap will clear all information it holds. > +* An incremental backup created from an empty bitmap will copy no data, > +as if nothing has changed. > + > +```json > +{ "execute": "block-dirty-bitmap-clear", > + "arguments": { > + "node": "drive0", > + "name": "bitmap0", > + } I'll quit pointing out broken trailing JSON commas, on the assumption that you'll fix all of them rather than stopping here :) > +} > +``` > + > +## Transactions > + > +### Justification > +Bitmaps can be safely modified when the VM is paused or halted by using > +the basic QMP commands. For instance, you might perform the following actions: > + > +1. Boot the VM in a paused state. > +2. Create a full drive backup of drive0 > +3. Create a new bitmap attached to drive0 > +4. resume execution of the VM > +5. Incremental backups are ready to be created. Consistency on using trailing '.'? > + > +At this point, the bitmap and drive backup would be correctly in sync, > +and incremental backups made from this point forward would be correctly aligned > +to the full drive backup. > + > +This is not particularly useful if we decide we want to start incremental > +backups after the VM has been running for a while, which will allow us to > +perform actions like the following: > + > +1. Boot the VM and begin execution > +2. Using transactions, perform the following operations: > + * Create 'bitmap0' > + * Create a full drive backup of drive0 > +3. Incremental backups are now ready to be created. It's unclear whether point 2 is describing two separate transactions, or one transaction with two steps. If it's just one, I'd word it: Using a single transaction, perform the following operations: > + > +The star of the show. > + > +**Nota Bene!** Only incremental backups of entire drives are supported for now. > +So despite the fact that you can attach a bitmap to any arbitrary node, they are > +only currently useful when attached to the root node. This is because > +drive-backup only supports drives/devices instead of arbitrary nodes. Maybe it's time to think about adding a node-backup :) But doesn't have to be this series.
On 02/20/2015 06:55 PM, Eric Blake wrote: > On 02/20/2015 04:07 PM, John Snow wrote: >> Signed-off-by: John Snow <jsnow@redhat.com> >> --- >> docs/bitmaps.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 253 insertions(+) >> create mode 100644 docs/bitmaps.md >> >> diff --git a/docs/bitmaps.md b/docs/bitmaps.md >> new file mode 100644 >> index 0000000..7cda146 >> --- /dev/null >> +++ b/docs/bitmaps.md >> @@ -0,0 +1,253 @@ >> +# Dirty Bitmaps > > No copyright/license? Might be good to add. > OK. > Also, I'm a fan of reviewing docs before code (does the design make > sense in isolation, and then did we implement it) rather than last in > the series (we may have faithfully documented our code, but that > includes baking in any design flaws that reviewers are now blind to > because of reading the code) > Sorry, I definitely did write this *after* and I selfishly put it at the end of the series to avoid disrupting the patch numbers from the previous revision. >> +* To create a new bitmap that tracks changes in 32KiB segments: >> + >> +```json >> +{ "execute": "block-dirty-bitmap-add", >> + "arguments": { >> + "node": "drive0", >> + "name": "bitmap0", >> + "granularity": "32768" > > s/"32768"/32768/ (we are using a JSON integer, not string) > >> + } >> +} >> +``` >> + >> +### Deletion >> + >> +* Can be performed on a disabled bitmap, but not a frozen one. >> + >> +* Because bitmaps are only unique to the node to which they are attached, >> +you must specify the node/drive name here, too. >> + >> +```json >> +{ "execute": "block-dirty-bitmap-remove", >> + "arguments": { >> + "node": "drive0", >> + "name": "bitmap0", >> + } > > No trailing commas in JSON. > >> +} >> +``` >> + >> +### Enable/Disable: >> + >> +* Not very useful in current cases, but potentially useful for debugging in the >> +future where we'd like to see what information changed only in a specific >> +time period: >> + >> +* To enable (which is, again, the default state after add) >> + >> +```json >> +{ "execute": "block-dirty-bitmap-enable", >> + "arguments": { >> + "node": "drive0", >> + "name": "bitmap0", >> + } > > and again. > >> +} >> +``` >> + >> +* To disable: >> + >> +```json >> +{ "execute": "block-dirty-bitmap-disable", >> + "arguments": { >> + "node": "drive0", >> + "name": "bitmap0", >> + } >> +} >> +``` > > and again. Also, maybe swap these two, since a bitmap defaults to > enabled (that is, you would logically use disable first, then enable to > re-enable, when testing these out, if there is no way to create an > already-disabled bitmap). > >> + >> +### Resetting >> + >> +* Resetting a bitmap will clear all information it holds. >> +* An incremental backup created from an empty bitmap will copy no data, >> +as if nothing has changed. >> + >> +```json >> +{ "execute": "block-dirty-bitmap-clear", >> + "arguments": { >> + "node": "drive0", >> + "name": "bitmap0", >> + } > > I'll quit pointing out broken trailing JSON commas, on the assumption > that you'll fix all of them rather than stopping here :) > Yeah, sorry! I have discovered today that I am *awful* at writing out json by hand. I promise the commands work otherwise! I will fix up the documentation here; if this winds up being the *ONLY* thing wrong with V14, I would prefer this doc be dropped from this series and I will just send a v2 for the doc by itself afterwards. >> +} >> +``` >> + >> +## Transactions >> + >> +### Justification >> +Bitmaps can be safely modified when the VM is paused or halted by using >> +the basic QMP commands. For instance, you might perform the following actions: >> + >> +1. Boot the VM in a paused state. >> +2. Create a full drive backup of drive0 >> +3. Create a new bitmap attached to drive0 >> +4. resume execution of the VM >> +5. Incremental backups are ready to be created. > > Consistency on using trailing '.'? > >> + >> +At this point, the bitmap and drive backup would be correctly in sync, >> +and incremental backups made from this point forward would be correctly aligned >> +to the full drive backup. >> + >> +This is not particularly useful if we decide we want to start incremental >> +backups after the VM has been running for a while, which will allow us to >> +perform actions like the following: >> + >> +1. Boot the VM and begin execution >> +2. Using transactions, perform the following operations: >> + * Create 'bitmap0' >> + * Create a full drive backup of drive0 >> +3. Incremental backups are now ready to be created. > > It's unclear whether point 2 is describing two separate transactions, or > one transaction with two steps. If it's just one, I'd word it: > > Using a single transaction, perform the following operations: > > OK. >> + >> +The star of the show. >> + >> +**Nota Bene!** Only incremental backups of entire drives are supported for now. >> +So despite the fact that you can attach a bitmap to any arbitrary node, they are >> +only currently useful when attached to the root node. This is because >> +drive-backup only supports drives/devices instead of arbitrary nodes. > > Maybe it's time to think about adding a node-backup :) But doesn't have > to be this series. > > I will crawl before I walk. And certainly before I start running and flying :) Thanks, --JS
diff --git a/docs/bitmaps.md b/docs/bitmaps.md new file mode 100644 index 0000000..7cda146 --- /dev/null +++ b/docs/bitmaps.md @@ -0,0 +1,253 @@ +# Dirty Bitmaps + +* Dirty bitmaps can be created at any time and attached to any node +(not just complete drives.) + +## Dirty Bitmap Names + +* A dirty bitmap's name is unique to the node, but bitmaps attached to different +nodes can share the same name. + +## Bitmap Modes + +* A Bitmap can be "enabled" (tracking writes, the default) or "disabled" +(read-only, I/O is ignored.) + +* A Bitmap can be "frozen," which means that it is currently in-use by a backup +operation and cannot be deleted, enabled, disabled, renamed, written to, reset, +etc. + +## Basic QMP Usage + +### Supported Commands ### + +* block-dirty-bitmap-add +* block-dirty-bitmap-remove +* block-dirty-bitmap-enable +* block-dirty-bitmap-disable +* block-dirty-bitmap-clear + +### Creation + +* To create a new bitmap, enabled, on the drive with id=drive0: + +```json +{ "execute": "block-dirty-bitmap-add", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } +} +``` + +* To create a new bitmap that tracks changes in 32KiB segments: + +```json +{ "execute": "block-dirty-bitmap-add", + "arguments": { + "node": "drive0", + "name": "bitmap0", + "granularity": "32768" + } +} +``` + +### Deletion + +* Can be performed on a disabled bitmap, but not a frozen one. + +* Because bitmaps are only unique to the node to which they are attached, +you must specify the node/drive name here, too. + +```json +{ "execute": "block-dirty-bitmap-remove", + "arguments": { + "node": "drive0", + "name": "bitmap0", + } +} +``` + +### Enable/Disable: + +* Not very useful in current cases, but potentially useful for debugging in the +future where we'd like to see what information changed only in a specific +time period: + +* To enable (which is, again, the default state after add) + +```json +{ "execute": "block-dirty-bitmap-enable", + "arguments": { + "node": "drive0", + "name": "bitmap0", + } +} +``` + +* To disable: + +```json +{ "execute": "block-dirty-bitmap-disable", + "arguments": { + "node": "drive0", + "name": "bitmap0", + } +} +``` + +### Resetting + +* Resetting a bitmap will clear all information it holds. +* An incremental backup created from an empty bitmap will copy no data, +as if nothing has changed. + +```json +{ "execute": "block-dirty-bitmap-clear", + "arguments": { + "node": "drive0", + "name": "bitmap0", + } +} +``` + +## Transactions + +### Justification +Bitmaps can be safely modified when the VM is paused or halted by using +the basic QMP commands. For instance, you might perform the following actions: + +1. Boot the VM in a paused state. +2. Create a full drive backup of drive0 +3. Create a new bitmap attached to drive0 +4. resume execution of the VM +5. Incremental backups are ready to be created. + +At this point, the bitmap and drive backup would be correctly in sync, +and incremental backups made from this point forward would be correctly aligned +to the full drive backup. + +This is not particularly useful if we decide we want to start incremental +backups after the VM has been running for a while, which will allow us to +perform actions like the following: + +1. Boot the VM and begin execution +2. Using transactions, perform the following operations: + * Create 'bitmap0' + * Create a full drive backup of drive0 +3. Incremental backups are now ready to be created. + +### Supported Bitmap Transactions + +* block-dirty-bitmap-add +* block-dirty-bitmap-enable +* block-dirty-bitmap-disable +* block-dirty-bitmap-clear + +The usages are identical to their respective QMP commands, but see below +for examples. + +### Example: New Incremental Backup + +As outlined in the justification, perhaps we want to create a new incremental +backup chain attached to a drive. + +```json +{ "execute": "transaction", + "arguments": { + "actions": [ + {"type": "block-dirty-bitmap-add", + "data": {"node": "drive0", "name": "bitmap0"} }, + {"type": "drive-backup", + "data": {"device": "drive0", "target": "/path/to/full_backup.img", + "sync": "full", "format": "qcow2"} } + ] + } +} +``` + +### Example: New Incremental Backup Anchor Point + +Maybe we just want to create a new full backup with an existing bitmap and +want to reset the bitmap to track the new chain. + +```json +{ "execute": "transaction", + "arguments": { + "actions": [ + {"type": "block-dirty-bitmap-clear", + "data": {"node": "drive0", "name": "bitmap0"} }, + {"type": "drive-backup", + "data": {"device": "drive0", "target": "/path/to/new_full_backup.img", + "sync": "full", "format": "qcow2"} } + ] + } +} +``` + +## Incremental Backups + +The star of the show. + +**Nota Bene!** Only incremental backups of entire drives are supported for now. +So despite the fact that you can attach a bitmap to any arbitrary node, they are +only currently useful when attached to the root node. This is because +drive-backup only supports drives/devices instead of arbitrary nodes. + +### Example: First Incremental Backup + +1. Create a full backup and sync it to the dirty bitmap, as in the transactional +examples above; or with the VM offline, manually create a full copy and then +create a new bitmap before the VM begins execution. + + * Let's assume the full backup is named 'full_backup.img.' + * Let's assume the bitmap you created is 'bitmap0' attached to 'drive0'. + +2. Create a destination image for the incremental backup that utilizes the +full backup as a backing image. + + * Let's assume it is named 'incremental.0.img' + +```sh +# qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 +``` + +3. Issue the incremental backup command: + +```json +{ "execute": "drive-backup", + "arguments": { + "device": "drive0", + "bitmap": "bitmap0", + "target": "incremental.0.img", + "format": "qcow2", + "sync": "dirty-bitmap", + "mode": "existing" + } +} +``` + +### Example: Second Incremental Backup + +1. Create a new destination image for the incremental backup that points to the +previous one, e.g.: 'incremental.1.img' + +```bash +# qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2 +``` + +2. Issue a new incremental backup command. The only difference here is that we +have changed the target image below. + +```json +{ "execute": "drive-backup", + "arguments": { + "device": "drive0", + "bitmap": "bitmap0", + "target": "incremental.1.img", + "format": "qcow2", + "sync": "dirty-bitmap", + "mode": "existing" + } +} +```
Signed-off-by: John Snow <jsnow@redhat.com> --- docs/bitmaps.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 docs/bitmaps.md