[v5,for,2.13,3/4] docs: tpm: add VM save/restore example and troubleshooting guide

Extend the docs related to TPM with specs related to VM save and
restore and a troubleshooting guide for TPM migration.

Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
---
 docs/specs/tpm.txt | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)

Hi

On Fri, Mar 16, 2018 at 10:46 PM, Stefan Berger
<stefanb@linux.vnet.ibm.com> wrote:
> Extend the docs related to TPM with specs related to VM save and
> restore and a troubleshooting guide for TPM migration.
>

Thanks a lot for writing this! some questions below

> Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
> ---
>  docs/specs/tpm.txt | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>
> diff --git a/docs/specs/tpm.txt b/docs/specs/tpm.txt
> index d1d7157..c230c4c 100644
> --- a/docs/specs/tpm.txt
> +++ b/docs/specs/tpm.txt
> @@ -200,3 +200,109 @@ crw-------. 1 root root 10, 224 Jul 11 10:11 /dev/tpm0
>  PCR-00: 35 4E 3B CE 23 9F 38 59 ...
>  ...
>  PCR-23: 00 00 00 00 00 00 00 00 ...
> +
> +
> +=== Migration with the TPM emulator ===
> +
> +The TPM emulator supports the following types of virtual machine migration:
> +
> +- VM save / restore (migration into a file)
> +- Network migration
> +- Snapshotting (migration into storage like QoW2 or QED)
> +
> +The following command sequences can be used to test VM save / restore.
> +
> +
> +In a 1st terminal start an instance of a swtpm using the following command:
> +
> +mkdir /tmp/mytpm1
> +swtpm socket --tpmstate dir=/tmp/mytpm1 \
> +  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
> +  --log level=20 --tpm2
> +
> +In a 2nd terminal start the VM:
> +
> +qemu-system-x86_64 -display sdl -enable-kvm \
> +  -m 1024 -boot d -bios bios-256k.bin -boot menu=on \
> +  -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
> +  -tpmdev emulator,id=tpm0,chardev=chrtpm \
> +  -device tpm-tis,tpmdev=tpm0 \
> +  -monitor stdio \
> +  test.img
> +
> +Verify that the attached TPM is working as expected using applications inside
> +the VM.
> +
> +To store the state of the VM use the following command in the QEMU monitor in
> +the 2nd terminal:
> +
> +(qemu) migrate "exec:cat > testvm.bin"
> +(qemu) quit
> +
> +At this point a file called 'testvm.bin' should exists and the swtpm and QEMU
> +processes should have ended.

When is swtpm ending, when qemu leaves? Hopefully you can do several
migrate commands.

> +
> +To test 'VM restore' you have to start the swtpm with the same parameters
> +as before. If previously a TPM 2 [--tpm2] was saved, --tpm2 must now be
> +passed again on the command line.
> +
> +In the 1st terminal restart the swtpm with the same command line as before:
> +
> +swtpm socket --tpmstate dir=/tmp/mytpm1 \
> +  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
> +  --log level=20 --tpm2

Does that mean the tpmstate directory content must be the same and
thus migrated as well? Can in be empty in the destination? If not,
what should be done to initialize it? Could it be empty instead?

> +
> +In the 2nd terminal restore the state of the VM using the additonal
> +'-incoming' option.
> +
> +qemu-system-x86_64 -display sdl -enable-kvm \
> +  -m 1024 -boot d -bios bios-256k.bin -boot menu=on \
> +  -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
> +  -tpmdev emulator,id=tpm0,chardev=chrtpm \
> +  -device tpm-tis,tpmdev=tpm0 \
> +  -incoming "exec:cat < testvm.bin" \
> +  test.img
> +
> +
> +Troubleshooting migration:
> +
> +There are several reasons why migration may fail. In case of problems,
> +please ensure that the command lines adhere to the following rules and,
> +if possible, that identical versions of QEMU and swtpm are used at all
> +times.
> +
> +VM save and restore:
> + - QEMU command line parameters should be identical apart from the
> +   '-incoming' option on VM restore
> + - swtpm command line parameters should be identical
> +
> +VM migration to 'localhost':
> + - QEMU command line parameters should be identical apart from the
> +   '-incoming' option on the destination side
> + - swtpm command line parameters should point to two different
> +   directories on the source and destination swtpm (--tpmstate dir=...)
> +   (especially if different versions of libtpms were to be used on the
> +   same machine).
> +
> +VM migration across the network:
> + - QEMU command line parameters should be identical apart from the
> +   '-incoming' option on the destination side
> + - swtpm command line parameters should be identical
> +
> +VM Snapshotting:
> + - QEMU command line parameters should be identical
> + - swtpm command line parameters should be identical
> +
> +
> +Besides that, migration failure reasons on the swtpm level may include
> +the following:
> +
> + - the versions of the swtpm on the source and destination sides are
> +   incompatible
> +   - downgrading of TPM state may not be supported
> +   - the source and destination libtpms were compiled with different
> +     compile-time options and the destination side refuses to accept the
> +     state
> + - different migration keys are used on the source and destination side
> +   and the destination side cannot decrypt the migrated state
> +   (swtpm ... --migration-key ... )
> --
> 2.5.5
>

On 03/28/2018 11:48 AM, Marc-André Lureau wrote:
> Hi
>
> On Fri, Mar 16, 2018 at 10:46 PM, Stefan Berger
> <stefanb@linux.vnet.ibm.com> wrote:
>> Extend the docs related to TPM with specs related to VM save and
>> restore and a troubleshooting guide for TPM migration.
>>
> Thanks a lot for writing this! some questions below
>
>> Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
>> ---
>>   docs/specs/tpm.txt | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 106 insertions(+)
>>
>> diff --git a/docs/specs/tpm.txt b/docs/specs/tpm.txt
>> index d1d7157..c230c4c 100644
>> --- a/docs/specs/tpm.txt
>> +++ b/docs/specs/tpm.txt
>> @@ -200,3 +200,109 @@ crw-------. 1 root root 10, 224 Jul 11 10:11 /dev/tpm0
>>   PCR-00: 35 4E 3B CE 23 9F 38 59 ...
>>   ...
>>   PCR-23: 00 00 00 00 00 00 00 00 ...
>> +
>> +
>> +=== Migration with the TPM emulator ===
>> +
>> +The TPM emulator supports the following types of virtual machine migration:
>> +
>> +- VM save / restore (migration into a file)
>> +- Network migration
>> +- Snapshotting (migration into storage like QoW2 or QED)
>> +
>> +The following command sequences can be used to test VM save / restore.
>> +
>> +
>> +In a 1st terminal start an instance of a swtpm using the following command:
>> +
>> +mkdir /tmp/mytpm1
>> +swtpm socket --tpmstate dir=/tmp/mytpm1 \
>> +  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
>> +  --log level=20 --tpm2
>> +
>> +In a 2nd terminal start the VM:
>> +
>> +qemu-system-x86_64 -display sdl -enable-kvm \
>> +  -m 1024 -boot d -bios bios-256k.bin -boot menu=on \
>> +  -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
>> +  -tpmdev emulator,id=tpm0,chardev=chrtpm \
>> +  -device tpm-tis,tpmdev=tpm0 \
>> +  -monitor stdio \
>> +  test.img
>> +
>> +Verify that the attached TPM is working as expected using applications inside
>> +the VM.
>> +
>> +To store the state of the VM use the following command in the QEMU monitor in
>> +the 2nd terminal:
>> +
>> +(qemu) migrate "exec:cat > testvm.bin"
>> +(qemu) quit
>> +
>> +At this point a file called 'testvm.bin' should exists and the swtpm and QEMU
>> +processes should have ended.
> When is swtpm ending, when qemu leaves? Hopefully you can do several
> migrate commands.

Yes, QEMU does not send it the signal to shut down. We can fall back to 
the source if the destination fails.

>
>> +
>> +To test 'VM restore' you have to start the swtpm with the same parameters
>> +as before. If previously a TPM 2 [--tpm2] was saved, --tpm2 must now be
>> +passed again on the command line.
>> +
>> +In the 1st terminal restart the swtpm with the same command line as before:
>> +
>> +swtpm socket --tpmstate dir=/tmp/mytpm1 \
>> +  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
>> +  --log level=20 --tpm2
> Does that mean the tpmstate directory content must be the same and
> thus migrated as well? Can in be empty in the destination? If not,
> what should be done to initialize it? Could it be empty instead?

QEMU migrates the state of the TPM with the CMD_GET_STATEBLOB to 
retrieve the state blobs and CMD_SET_STATEBLOB to set them on the 
destination. The destination only needs to have the TPM running but the 
directory must have been created (--tpmstate dir=...).

One can try this with localhost migration over the network as well, but 
I didn't want to show this more complicated scenario in the doc:

destination QEMU:
sudo ./x86_64-softmmu/qemu-system-x86_64 -vnc :11 -enable-kvm -m 1024 
-smp 8 -boot d -L /usr/share/seabios -bios bios-256k.bin -boot menu=on 
-chardev socket,id=chrtpm,path=/tmp/mytpm2/ctrl.sock -tpmdev 
emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -monitor 
stdio -chardev file,id=pts2,path=/tmp/seabios.log -device 
isa-serial,chardev=pts2 /var/lib/libvirt/images/FC27 -incoming "exec:nc 
-l 127.0.0.1 12345"

source QEMU:
sudo ./x86_64-softmmu/qemu-system-x86_64 -vnc :10 -enable-kvm -m 1024 
-smp 8 -boot d -L /usr/share/seabios -bios bios-256k.bin -boot menu=on 
-chardev socket,id=chrtpm,path=/tmp/mytpm1/ctrl.sock -tpmdev 
emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -monitor 
stdio -chardev file,id=pts2,path=/tmp/seabios.log -device 
isa-serial,chardev=pts2  /var/lib/libvirt/images/FC27

(qemu) migrate "exec:nc 127.0.0.1 12345"

Just tested again.

     Stefan


>> +
>> +In the 2nd terminal restore the state of the VM using the additonal
>> +'-incoming' option.
>> +
>> +qemu-system-x86_64 -display sdl -enable-kvm \
>> +  -m 1024 -boot d -bios bios-256k.bin -boot menu=on \
>> +  -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
>> +  -tpmdev emulator,id=tpm0,chardev=chrtpm \
>> +  -device tpm-tis,tpmdev=tpm0 \
>> +  -incoming "exec:cat < testvm.bin" \
>> +  test.img
>> +
>> +
>> +Troubleshooting migration:
>> +
>> +There are several reasons why migration may fail. In case of problems,
>> +please ensure that the command lines adhere to the following rules and,
>> +if possible, that identical versions of QEMU and swtpm are used at all
>> +times.
>> +
>> +VM save and restore:
>> + - QEMU command line parameters should be identical apart from the
>> +   '-incoming' option on VM restore
>> + - swtpm command line parameters should be identical
>> +
>> +VM migration to 'localhost':
>> + - QEMU command line parameters should be identical apart from the
>> +   '-incoming' option on the destination side
>> + - swtpm command line parameters should point to two different
>> +   directories on the source and destination swtpm (--tpmstate dir=...)
>> +   (especially if different versions of libtpms were to be used on the
>> +   same machine).
>> +
>> +VM migration across the network:
>> + - QEMU command line parameters should be identical apart from the
>> +   '-incoming' option on the destination side
>> + - swtpm command line parameters should be identical
>> +
>> +VM Snapshotting:
>> + - QEMU command line parameters should be identical
>> + - swtpm command line parameters should be identical
>> +
>> +
>> +Besides that, migration failure reasons on the swtpm level may include
>> +the following:
>> +
>> + - the versions of the swtpm on the source and destination sides are
>> +   incompatible
>> +   - downgrading of TPM state may not be supported
>> +   - the source and destination libtpms were compiled with different
>> +     compile-time options and the destination side refuses to accept the
>> +     state
>> + - different migration keys are used on the source and destination side
>> +   and the destination side cannot decrypt the migrated state
>> +   (swtpm ... --migration-key ... )
>> --
>> 2.5.5
>>
>
>
>

* Stefan Berger (stefanb@linux.vnet.ibm.com) wrote:
> On 03/28/2018 11:48 AM, Marc-André Lureau wrote:
> > Hi
> > 
> > On Fri, Mar 16, 2018 at 10:46 PM, Stefan Berger
> > <stefanb@linux.vnet.ibm.com> wrote:
> > > Extend the docs related to TPM with specs related to VM save and
> > > restore and a troubleshooting guide for TPM migration.
> > > 
> > Thanks a lot for writing this! some questions below
> > 
> > > Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
> > > ---
> > >   docs/specs/tpm.txt | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >   1 file changed, 106 insertions(+)
> > > 
> > > diff --git a/docs/specs/tpm.txt b/docs/specs/tpm.txt
> > > index d1d7157..c230c4c 100644
> > > --- a/docs/specs/tpm.txt
> > > +++ b/docs/specs/tpm.txt
> > > @@ -200,3 +200,109 @@ crw-------. 1 root root 10, 224 Jul 11 10:11 /dev/tpm0
> > >   PCR-00: 35 4E 3B CE 23 9F 38 59 ...
> > >   ...
> > >   PCR-23: 00 00 00 00 00 00 00 00 ...
> > > +
> > > +
> > > +=== Migration with the TPM emulator ===
> > > +
> > > +The TPM emulator supports the following types of virtual machine migration:
> > > +
> > > +- VM save / restore (migration into a file)
> > > +- Network migration
> > > +- Snapshotting (migration into storage like QoW2 or QED)
> > > +
> > > +The following command sequences can be used to test VM save / restore.
> > > +
> > > +
> > > +In a 1st terminal start an instance of a swtpm using the following command:
> > > +
> > > +mkdir /tmp/mytpm1
> > > +swtpm socket --tpmstate dir=/tmp/mytpm1 \
> > > +  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
> > > +  --log level=20 --tpm2
> > > +
> > > +In a 2nd terminal start the VM:
> > > +
> > > +qemu-system-x86_64 -display sdl -enable-kvm \
> > > +  -m 1024 -boot d -bios bios-256k.bin -boot menu=on \
> > > +  -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
> > > +  -tpmdev emulator,id=tpm0,chardev=chrtpm \
> > > +  -device tpm-tis,tpmdev=tpm0 \
> > > +  -monitor stdio \
> > > +  test.img
> > > +
> > > +Verify that the attached TPM is working as expected using applications inside
> > > +the VM.
> > > +
> > > +To store the state of the VM use the following command in the QEMU monitor in
> > > +the 2nd terminal:
> > > +
> > > +(qemu) migrate "exec:cat > testvm.bin"
> > > +(qemu) quit
> > > +
> > > +At this point a file called 'testvm.bin' should exists and the swtpm and QEMU
> > > +processes should have ended.
> > When is swtpm ending, when qemu leaves? Hopefully you can do several
> > migrate commands.
> 
> Yes, QEMU does not send it the signal to shut down. We can fall back to the
> source if the destination fails.
> 
> > 
> > > +
> > > +To test 'VM restore' you have to start the swtpm with the same parameters
> > > +as before. If previously a TPM 2 [--tpm2] was saved, --tpm2 must now be
> > > +passed again on the command line.
> > > +
> > > +In the 1st terminal restart the swtpm with the same command line as before:
> > > +
> > > +swtpm socket --tpmstate dir=/tmp/mytpm1 \
> > > +  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
> > > +  --log level=20 --tpm2
> > Does that mean the tpmstate directory content must be the same and
> > thus migrated as well? Can in be empty in the destination? If not,
> > what should be done to initialize it? Could it be empty instead?
> 
> QEMU migrates the state of the TPM with the CMD_GET_STATEBLOB to retrieve
> the state blobs and CMD_SET_STATEBLOB to set them on the destination. The
> destination only needs to have the TPM running but the directory must have
> been created (--tpmstate dir=...).
> 
> One can try this with localhost migration over the network as well, but I
> didn't want to show this more complicated scenario in the doc:
> 
> destination QEMU:
> sudo ./x86_64-softmmu/qemu-system-x86_64 -vnc :11 -enable-kvm -m 1024 -smp 8
> -boot d -L /usr/share/seabios -bios bios-256k.bin -boot menu=on -chardev
> socket,id=chrtpm,path=/tmp/mytpm2/ctrl.sock -tpmdev
> emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -monitor stdio
> -chardev file,id=pts2,path=/tmp/seabios.log -device isa-serial,chardev=pts2
> /var/lib/libvirt/images/FC27 -incoming "exec:nc -l 127.0.0.1 12345"
> 
> source QEMU:
> sudo ./x86_64-softmmu/qemu-system-x86_64 -vnc :10 -enable-kvm -m 1024 -smp 8
> -boot d -L /usr/share/seabios -bios bios-256k.bin -boot menu=on -chardev
> socket,id=chrtpm,path=/tmp/mytpm1/ctrl.sock -tpmdev
> emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -monitor stdio
> -chardev file,id=pts2,path=/tmp/seabios.log -device isa-serial,chardev=pts2
> /var/lib/libvirt/images/FC27
> 
> (qemu) migrate "exec:nc 127.0.0.1 12345"

Migration has a tcp form that's a bit easier;
-incoming tcp:127.0.0.1:12345
migrate tcp:127.0.0.1:12345

Dave

> Just tested again.
> 
>     Stefan
> 
> 
> > > +
> > > +In the 2nd terminal restore the state of the VM using the additonal
> > > +'-incoming' option.
> > > +
> > > +qemu-system-x86_64 -display sdl -enable-kvm \
> > > +  -m 1024 -boot d -bios bios-256k.bin -boot menu=on \
> > > +  -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
> > > +  -tpmdev emulator,id=tpm0,chardev=chrtpm \
> > > +  -device tpm-tis,tpmdev=tpm0 \
> > > +  -incoming "exec:cat < testvm.bin" \
> > > +  test.img
> > > +
> > > +
> > > +Troubleshooting migration:
> > > +
> > > +There are several reasons why migration may fail. In case of problems,
> > > +please ensure that the command lines adhere to the following rules and,
> > > +if possible, that identical versions of QEMU and swtpm are used at all
> > > +times.
> > > +
> > > +VM save and restore:
> > > + - QEMU command line parameters should be identical apart from the
> > > +   '-incoming' option on VM restore
> > > + - swtpm command line parameters should be identical
> > > +
> > > +VM migration to 'localhost':
> > > + - QEMU command line parameters should be identical apart from the
> > > +   '-incoming' option on the destination side
> > > + - swtpm command line parameters should point to two different
> > > +   directories on the source and destination swtpm (--tpmstate dir=...)
> > > +   (especially if different versions of libtpms were to be used on the
> > > +   same machine).
> > > +
> > > +VM migration across the network:
> > > + - QEMU command line parameters should be identical apart from the
> > > +   '-incoming' option on the destination side
> > > + - swtpm command line parameters should be identical
> > > +
> > > +VM Snapshotting:
> > > + - QEMU command line parameters should be identical
> > > + - swtpm command line parameters should be identical
> > > +
> > > +
> > > +Besides that, migration failure reasons on the swtpm level may include
> > > +the following:
> > > +
> > > + - the versions of the swtpm on the source and destination sides are
> > > +   incompatible
> > > +   - downgrading of TPM state may not be supported
> > > +   - the source and destination libtpms were compiled with different
> > > +     compile-time options and the destination side refuses to accept the
> > > +     state
> > > + - different migration keys are used on the source and destination side
> > > +   and the destination side cannot decrypt the migrated state
> > > +   (swtpm ... --migration-key ... )
> > > --
> > > 2.5.5
> > > 
> > 
> > 
> > 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK