# [v2] virtio-blk physical block size

Message ID 1262018363-15871-1-git-send-email-avi@redhat.com New show

## Commit Message

Avi Kivity Dec. 28, 2009, 4:39 p.m.
This patch adds a physical block size attribute to virtio disks,
corresponding to /sys/devices/.../physical_block_size.  It is defined as
the request alignment which will not trigger RMW cycles.  This can be
important for modern disks which use 4K physical sectors (though they
still support 512 logical sectors), and for file-backed disk images (which
have both the underlying filesystem block size and their own allocation
granularity to consider).

Installers use this to align partitions to physical block boundaries.

Note the spec already defined blk_size as the performance rather than
minimum alignment.  However the driver interpreted this as the logical
block size, so I updated the spec to match the driver assuming the driver
predates the spec and that this is an error.

Signed-off-by: Avi Kivity <avi@redhat.com>

--

v2: feature bit 7 was already used by an undocumented feature; use a new bit


Rusty Russell Jan. 4, 2010, 3:08 a.m. | #1
On Tue, 29 Dec 2009 03:09:23 am Avi Kivity wrote:
> This patch adds a physical block size attribute to virtio disks,
> corresponding to /sys/devices/.../physical_block_size.  It is defined as
> the request alignment which will not trigger RMW cycles.  This can be
> important for modern disks which use 4K physical sectors (though they
> still support 512 logical sectors), and for file-backed disk images (which
> have both the underlying filesystem block size and their own allocation
> granularity to consider).
>
> Installers use this to align partitions to physical block boundaries.
>
> Note the spec already defined blk_size as the performance rather than
> minimum alignment.  However the driver interpreted this as the logical
> block size, so I updated the spec to match the driver assuming the driver
> predates the spec and that this is an error.

I thought this was what I was doing, but I have shown over and over that
I have no idea about block devices.

Our current driver treats BLK_SIZE as the logical and physical size (see
blk_queue_logical_block_size).

I have no idea what "logical" vs. "physical" actually means.  Anyone?  Most
importantly, is it some Linux-internal difference or a real I/O-visible
distinction?

Rusty.

Avi Kivity Jan. 4, 2010, 7:02 a.m. | #2
On 01/04/2010 05:08 AM, Rusty Russell wrote:
> On Tue, 29 Dec 2009 03:09:23 am Avi Kivity wrote:
>
>> This patch adds a physical block size attribute to virtio disks,
>> corresponding to /sys/devices/.../physical_block_size.  It is defined as
>> the request alignment which will not trigger RMW cycles.  This can be
>> important for modern disks which use 4K physical sectors (though they
>> still support 512 logical sectors), and for file-backed disk images (which
>> have both the underlying filesystem block size and their own allocation
>> granularity to consider).
>>
>> Installers use this to align partitions to physical block boundaries.
>>
>> Note the spec already defined blk_size as the performance rather than
>> minimum alignment.  However the driver interpreted this as the logical
>> block size, so I updated the spec to match the driver assuming the driver
>> predates the spec and that this is an error.
>>
> I thought this was what I was doing, but I have shown over and over that
> I have no idea about block devices.
>
> Our current driver treats BLK_SIZE as the logical and physical size (see
> blk_queue_logical_block_size).
>

But we want them to be different.

> I have no idea what "logical" vs. "physical" actually means.  Anyone?  Most
> importantly, is it some Linux-internal difference or a real I/O-visible
> distinction?
>

Yes.

Logical block size is the minimum block size the hardware will allow.
Try to write less than that, and the hardware will laugh in your face.

Physical block size is the what the logical block size would have been
is software didn't suck.  In theory they should be the same, but since
compatibility reaons clamp the logical block size to 512, they have to
differ.  A disk may have a physical block size of 4096 and emulate
logical block size of 512 on top of that using read-modify-write.

Or so I understand it.

Christoph Hellwig Jan. 4, 2010, 8:30 a.m. | #3
On Mon, Jan 04, 2010 at 01:38:51PM +1030, Rusty Russell wrote:
> I thought this was what I was doing, but I have shown over and over that
> I have no idea about block devices.
>
> Our current driver treats BLK_SIZE as the logical and physical size (see
> blk_queue_logical_block_size).
>
> I have no idea what "logical" vs. "physical" actually means.  Anyone?  Most
> importantly, is it some Linux-internal difference or a real I/O-visible
> distinction?

Those should be the same for any sane interface.  They are for classical
disk devices with larger block sizes (MO, s390 dasd) and also for the
now appearing 4k sector scsi disks.  But in the ide world people are
concerned about dos/window legacy compatiblity so they came up with a
nasty hack:

- there is a physical block size as used by the disk internally
(4k initially)
- all the interfaces to the operating system still happen in the
traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger
physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional
DOS partitions tables, so that the 512 byte block zero might even
have an offset into the first larger physical block.  This is also
exposed in the ATA identify information.

All in all I don't think this mess is a good idea to replicate in
virtio.  Virtio by defintion requires virtualization aware guests, so we
should just follow the SCSI way of larger real block sizes here.

>
> Rusty.
>
---end quoted text---

Rusty Russell Jan. 5, 2010, 12:56 p.m. | #4
On Mon, 4 Jan 2010 07:00:35 pm Christoph Hellwig wrote:
> On Mon, Jan 04, 2010 at 01:38:51PM +1030, Rusty Russell wrote:
> > I thought this was what I was doing, but I have shown over and over that
> > I have no idea about block devices.
> >
> > Our current driver treats BLK_SIZE as the logical and physical size (see
> > blk_queue_logical_block_size).
> >
> > I have no idea what "logical" vs. "physical" actually means.  Anyone?  Most
> > importantly, is it some Linux-internal difference or a real I/O-visible
> > distinction?
>
> Those should be the same for any sane interface.  They are for classical
> disk devices with larger block sizes (MO, s390 dasd) and also for the
> now appearing 4k sector scsi disks.  But in the ide world people are
> concerned about dos/window legacy compatiblity so they came up with a
> nasty hack:
>
>  - there is a physical block size as used by the disk internally
>    (4k initially)
>  - all the interfaces to the operating system still happen in the
>    traditional 512 byte blocks to not break any existing assumptions
>  - to make sure modern operating systems can optimize for the larger
>    physical sectors the disks expose this size, too.
>  - even worse disks can also have alignment hacks for the traditional
>    DOS partitions tables, so that the 512 byte block zero might even
>    have an offset into the first larger physical block.  This is also
>    exposed in the ATA identify information.
>
> All in all I don't think this mess is a good idea to replicate in
> virtio.  Virtio by defintion requires virtualization aware guests, so we
> should just follow the SCSI way of larger real block sizes here.

Yes.  The current VIRTIO_BLK_F_BLK_SIZE says "please use this block size".
We haven't actually specified what happens if the guest doesn't, but the
spec says "must", and the Linux implementation does so AFAICT.

If we want a "soft" size, we could add that as a separate feature.

Cheers,
Rusty.

Avi Kivity Jan. 5, 2010, 12:58 p.m. | #5
On 01/05/2010 02:56 PM, Rusty Russell wrote:
>
>> Those should be the same for any sane interface.  They are for classical
>> disk devices with larger block sizes (MO, s390 dasd) and also for the
>> now appearing 4k sector scsi disks.  But in the ide world people are
>> concerned about dos/window legacy compatiblity so they came up with a
>> nasty hack:
>>
>>   - there is a physical block size as used by the disk internally
>>     (4k initially)
>>   - all the interfaces to the operating system still happen in the
>>     traditional 512 byte blocks to not break any existing assumptions
>>   - to make sure modern operating systems can optimize for the larger
>>     physical sectors the disks expose this size, too.
>>   - even worse disks can also have alignment hacks for the traditional
>>     DOS partitions tables, so that the 512 byte block zero might even
>>     have an offset into the first larger physical block.  This is also
>>     exposed in the ATA identify information.
>>
>> All in all I don't think this mess is a good idea to replicate in
>> virtio.  Virtio by defintion requires virtualization aware guests, so we
>> should just follow the SCSI way of larger real block sizes here.
>>
> Yes.  The current VIRTIO_BLK_F_BLK_SIZE says "please use this block size".
> We haven't actually specified what happens if the guest doesn't, but the
> spec says "must", and the Linux implementation does so AFAICT.
>
> If we want a "soft" size, we could add that as a separate feature.
>

No - I agree with Christoph, there's no reason to use a 512/4096
monstrosity with virtio.

Jamie Lokier Jan. 5, 2010, 8:16 p.m. | #6
Avi Kivity wrote:
> On 01/05/2010 02:56 PM, Rusty Russell wrote:
> >
> >>Those should be the same for any sane interface.  They are for classical
> >>disk devices with larger block sizes (MO, s390 dasd) and also for the
> >>now appearing 4k sector scsi disks.  But in the ide world people are
> >>concerned about dos/window legacy compatiblity so they came up with a
> >>nasty hack:
> >>
> >>  - there is a physical block size as used by the disk internally
> >>    (4k initially)
> >>  - all the interfaces to the operating system still happen in the
> >>    traditional 512 byte blocks to not break any existing assumptions
> >>  - to make sure modern operating systems can optimize for the larger
> >>    physical sectors the disks expose this size, too.
> >>  - even worse disks can also have alignment hacks for the traditional
> >>    DOS partitions tables, so that the 512 byte block zero might even
> >>    have an offset into the first larger physical block.  This is also
> >>    exposed in the ATA identify information.
> >>
> >>All in all I don't think this mess is a good idea to replicate in
> >>virtio.  Virtio by defintion requires virtualization aware guests, so we
> >>should just follow the SCSI way of larger real block sizes here.
> >>
> >Yes.  The current VIRTIO_BLK_F_BLK_SIZE says "please use this block size".
> >We haven't actually specified what happens if the guest doesn't, but the
> >spec says "must", and the Linux implementation does so AFAICT.
> >
> >If we want a "soft" size, we could add that as a separate feature.
> >
>
> No - I agree with Christoph, there's no reason to use a 512/4096
> monstrosity with virtio.

It would be good if virtio relayed the backing device's basic topology
hints, so:

- If the backing dev is a real disk with 512-byte sectors,
virtio should indicate 512-byte blocks to the guest.

- If the backing dev is a real disk with 4096-byte sectors,
virtio should indicate 4096-byte blocks to the guest.

With databases and filesystems, if you care about data integrity:

- If the backing dev is a real disk with 4096-byte sectors,
or a file whose access is through a 4096-byte-per-page cache,
virtio must indicate 4096-byte blocks otherwise guest
journalling is not host-powerfail safe.

You get the idea.  If there is only one parameter, it really should be
at least as large as the smallest unit which may be corrupted by
writes when errors occur.

-- Jamie

Jamie Lokier Jan. 5, 2010, 8:18 p.m. | #7
Avi Kivity wrote:
> Physical block size is the what the logical block size would have been
> is software didn't suck.  In theory they should be the same, but since
> compatibility reaons clamp the logical block size to 512, they have to
> differ.  A disk may have a physical block size of 4096 and emulate
> logical block size of 512 on top of that using read-modify-write.
>
> Or so I understand it.

I think that's right, but a side effect is that if you get a power
failure during the read-modify-write, bytes anywhere in 4096 sector
may be incorrect, so journalling (etc.) needs to use 4096 byte blocks
for data integrity, even though the drive emulates smaller writes.

-- Jamie

Christoph Hellwig Jan. 8, 2010, 3:40 p.m. | #8
On Tue, Jan 05, 2010 at 08:16:15PM +0000, Jamie Lokier wrote:
> It would be good if virtio relayed the backing device's basic topology
> hints, so:
>
>     - If the backing dev is a real disk with 512-byte sectors,
>       virtio should indicate 512-byte blocks to the guest.
>
>     - If the backing dev is a real disk with 4096-byte sectors,
>       virtio should indicate 4096-byte blocks to the guest.
>
> With databases and filesystems, if you care about data integrity:
>
>     - If the backing dev is a real disk with 4096-byte sectors,
>       or a file whose access is through a 4096-byte-per-page cache,
>       virtio must indicate 4096-byte blocks otherwise guest
>       journalling is not host-powerfail safe.
>
> You get the idea.  If there is only one parameter, it really should be
> at least as large as the smallest unit which may be corrupted by
> writes when errors occur.

It's not that easy.  IDE only supports larger sectors sizes with the
physical sector size attribute, not native larger sectors.  While scsi
does support it it's untypical and I would not expect the guests to
not always get it right.  So the best is to use the transport native
way to express that we have larger sectors and expect the guest to do
the right thing.

I've done some work on the autodetection of the larger sector sizes
a while ago.  But now that people brought up migration I wonder if that
makes sense - if we migrate from a 512 byte sector size disk to a 4096
byte sector size disk we can't simply change guest visible attributes.

Maybe we should pick one on image creation and then stick to it.  For an
image format we could write down this information in the image, but for
a raw images that's impossible.

Avi Kivity Jan. 10, 2010, 12:35 p.m. | #9
On 01/08/2010 05:40 PM, Christoph Hellwig wrote:
> Maybe we should pick one on image creation and then stick to it.  For an
> image format we could write down this information in the image, but for
> a raw images that's impossible.
>

The management system should remember it (like it remembers which images
belong to which guest, and how to expose them).


## Patch

--- virtio-spec-0.8.3.lyx.orig	2009-12-28 17:42:40.000000000 +0200
+++ virtio-spec-0.8.3.lyx	2009-12-28 18:37:39.000000000 +0200
@@ -4138,6 +4138,19 @@
(10) Maximum total sectors in an I/O.
\end_layout

+\begin_layout Description
+VIRTIO_BLK_F_PHYS_BLK_SIZE(11) Physical block size of disk (writes aligned
+ to this will not trigger read-modify-write cycles) is in
+\begin_inset Quotes erd
+\end_inset
+
+phys_blk_size
+\begin_inset Quotes erd
+\end_inset
+
+.
+\end_layout
+
\end_deeper
\begin_layout Description
Device
@@ -4214,6 +4227,11 @@

\begin_layout Plain Layout

+    u32 phys_blk_size;
+\end_layout
+
+\begin_layout Plain Layout
+
};
\end_layout

@@ -4241,9 +4259,9 @@

\begin_layout Enumerate
If the VIRTIO_BLK_F_BLK_SIZE feature is negotiated, the blk_size field can
- be read to determine the optimal sector size for the driver to use.
+ be read to determine the sector size for the driver to use.
This does not effect the units used in the protocol (always 512 bytes),
- but awareness of the correct value can effect performance.
+ but requests must be aligned to this size.
\end_layout

\begin_layout Enumerate
@@ -4257,6 +4275,13 @@
No requests should be submitted which go beyond this limit.
\end_layout

+\begin_layout Enumerate
+If the VIRTIO_BLK_F_PHYS_BLK_SIZE feature is negotiated, the phys_blk_size
+ field should be read to determine the hardware block size.
+ Smaller writes are liable to require read-modify-write cycles on behalf
+ of the underlying hardware, however they are still supported.
+\end_layout
+
\begin_layout Section*
Device Operation
\end_layout