[v1] vnc: fix VNC artifacts
diff mbox series

Message ID 7cb1f22ef8d575238c84f091f03dee8e7fedb78d.1579232679.git.dirty@apple.com
State New
Headers show
Series
  • [v1] vnc: fix VNC artifacts
Related show

Commit Message

Keith Packard" via Jan. 17, 2020, 3:50 a.m. UTC
Remove VNC optimization to reencode framebuffer update as raw if it's
smaller than the default encoding.  QEMU's implementation was naive and
didn't account for the ZLIB z_stream mutating with each compression.  Just
saving and restoring the output buffer offset wasn't sufficient to "rewind"
the previous encoding.  Considering that ZRLE is never larger than raw and
even though ZLIB can occasionally be fractionally larger than raw, the
overhead of implementing this optimization correctly isn't worth it.

In my investigation, ZRLE always compresses better than ZLIB so
prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
preferred.

Fixes: <de3f7de7f4e257> ("vnc: allow fall back to RAW encoding")
Signed-off-by: Cameron Esfahani <dirty@apple.com>
---
 ui/vnc-enc-zrle.c |  4 ++--
 ui/vnc.c          | 30 +++++++++++-------------------
 2 files changed, 13 insertions(+), 21 deletions(-)

Comments

Gerd Hoffmann Jan. 17, 2020, 7:45 a.m. UTC | #1
On Thu, Jan 16, 2020 at 07:50:58PM -0800, Cameron Esfahani wrote:
> Remove VNC optimization to reencode framebuffer update as raw if it's
> smaller than the default encoding.  QEMU's implementation was naive and
> didn't account for the ZLIB z_stream mutating with each compression.  Just
> saving and restoring the output buffer offset wasn't sufficient to "rewind"
> the previous encoding.  Considering that ZRLE is never larger than raw and
> even though ZLIB can occasionally be fractionally larger than raw, the
> overhead of implementing this optimization correctly isn't worth it.

So just revert de3f7de7f4e257 then ...

> In my investigation, ZRLE always compresses better than ZLIB so
> prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
> preferred.

... and make this a separate patch?

cheers,
  Gerd
Keith Packard" via Jan. 17, 2020, 7:48 a.m. UTC | #2
Yes.  Personally, I'd also take the change to vnc-enc-zrle.c: because vs->zrle->zlib is reset at the top of the function, using vs->zrle->zlib.offset in determining zstream->next_out and zstream->avail_out is useless.

Cameron Esfahani
dirty@apple.com

"All that is necessary for the triumph of evil is that good men do nothing."

Edmund Burke



> On Jan 16, 2020, at 11:45 PM, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> On Thu, Jan 16, 2020 at 07:50:58PM -0800, Cameron Esfahani wrote:
>> Remove VNC optimization to reencode framebuffer update as raw if it's
>> smaller than the default encoding.  QEMU's implementation was naive and
>> didn't account for the ZLIB z_stream mutating with each compression.  Just
>> saving and restoring the output buffer offset wasn't sufficient to "rewind"
>> the previous encoding.  Considering that ZRLE is never larger than raw and
>> even though ZLIB can occasionally be fractionally larger than raw, the
>> overhead of implementing this optimization correctly isn't worth it.
> 
> So just revert de3f7de7f4e257 then ...
> 
>> In my investigation, ZRLE always compresses better than ZLIB so
>> prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
>> preferred.
> 
> ... and make this a separate patch?
> 
> cheers,
>  Gerd
>
Keith Packard" via Jan. 17, 2020, 9:43 p.m. UTC | #3
I’m new to this process, what are the next steps?

Cameron Esfahani
dirty@apple.com

> On Jan 16, 2020, at 11:47 PM, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> On Thu, Jan 16, 2020 at 07:50:58PM -0800, Cameron Esfahani wrote:
>> Remove VNC optimization to reencode framebuffer update as raw if it's
>> smaller than the default encoding.  QEMU's implementation was naive and
>> didn't account for the ZLIB z_stream mutating with each compression.  Just
>> saving and restoring the output buffer offset wasn't sufficient to "rewind"
>> the previous encoding.  Considering that ZRLE is never larger than raw and
>> even though ZLIB can occasionally be fractionally larger than raw, the
>> overhead of implementing this optimization correctly isn't worth it.
> 
> So just revert de3f7de7f4e257 then ...
> 
>> In my investigation, ZRLE always compresses better than ZLIB so
>> prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
>> preferred.
> 
> ... and make this a separate patch?
> 
> cheers,
>  Gerd
> 
>
Philippe Mathieu-Daudé Jan. 18, 2020, 6:24 a.m. UTC | #4
On 1/17/20 10:43 PM, Cameron Esfahani via wrote:
> I’m new to this process, what are the next steps?

Assuming your patch is in a branch named vncfix_v1:

1/ Start new branch based on the commit previous to your patch:

   - git checkout -b vncfix_v2 vncfix_v1~

2/ Revert the offending patch, explain in commit description why:

   - git revert de3f7de7f4e257

3/ Apply the rest of your patch on top, git-cherry-pick is smart to 
directly use the diff context. Verify the patch is correct and rewrite 
the commit description:

   - git cherry-pick vncfix_v1

4/ Send the 2 patches as a series to the mailing list

> 
> Cameron Esfahani
> dirty@apple.com
> 
>> On Jan 16, 2020, at 11:47 PM, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>
>> On Thu, Jan 16, 2020 at 07:50:58PM -0800, Cameron Esfahani wrote:
>>> Remove VNC optimization to reencode framebuffer update as raw if it's
>>> smaller than the default encoding.  QEMU's implementation was naive and
>>> didn't account for the ZLIB z_stream mutating with each compression.  Just
>>> saving and restoring the output buffer offset wasn't sufficient to "rewind"
>>> the previous encoding.  Considering that ZRLE is never larger than raw and
>>> even though ZLIB can occasionally be fractionally larger than raw, the
>>> overhead of implementing this optimization correctly isn't worth it.
>>
>> So just revert de3f7de7f4e257 then ...
>>
>>> In my investigation, ZRLE always compresses better than ZLIB so
>>> prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
>>> preferred.
>>
>> ... and make this a separate patch?
>>
>> cheers,
>>   Gerd
>>
>>
>

Patch
diff mbox series

diff --git a/ui/vnc-enc-zrle.c b/ui/vnc-enc-zrle.c
index 17fd28a2e2..b4f71e32cf 100644
--- a/ui/vnc-enc-zrle.c
+++ b/ui/vnc-enc-zrle.c
@@ -98,8 +98,8 @@  static int zrle_compress_data(VncState *vs, int level)
     /* set pointers */
     zstream->next_in = vs->zrle->zrle.buffer;
     zstream->avail_in = vs->zrle->zrle.offset;
-    zstream->next_out = vs->zrle->zlib.buffer + vs->zrle->zlib.offset;
-    zstream->avail_out = vs->zrle->zlib.capacity - vs->zrle->zlib.offset;
+    zstream->next_out = vs->zrle->zlib.buffer;
+    zstream->avail_out = vs->zrle->zlib.capacity;
     zstream->data_type = Z_BINARY;
 
     /* start encoding */
diff --git a/ui/vnc.c b/ui/vnc.c
index 4100d6e404..f085e1b747 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -898,8 +898,6 @@  int vnc_raw_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
 int vnc_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
 {
     int n = 0;
-    bool encode_raw = false;
-    size_t saved_offs = vs->output.offset;
 
     switch(vs->vnc_encoding) {
         case VNC_ENCODING_ZLIB:
@@ -922,24 +920,11 @@  int vnc_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
             n = vnc_zywrle_send_framebuffer_update(vs, x, y, w, h);
             break;
         default:
-            encode_raw = true;
+            vnc_framebuffer_update(vs, x, y, w, h, VNC_ENCODING_RAW);
+            n = vnc_raw_send_framebuffer_update(vs, x, y, w, h);
             break;
     }
 
-    /* If the client has the same pixel format as our internal buffer and
-     * a RAW encoding would need less space fall back to RAW encoding to
-     * save bandwidth and processing power in the client. */
-    if (!encode_raw && vs->write_pixels == vnc_write_pixels_copy &&
-        12 + h * w * VNC_SERVER_FB_BYTES <= (vs->output.offset - saved_offs)) {
-        vs->output.offset = saved_offs;
-        encode_raw = true;
-    }
-
-    if (encode_raw) {
-        vnc_framebuffer_update(vs, x, y, w, h, VNC_ENCODING_RAW);
-        n = vnc_raw_send_framebuffer_update(vs, x, y, w, h);
-    }
-
     return n;
 }
 
@@ -2087,8 +2072,15 @@  static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
             break;
 #endif
         case VNC_ENCODING_ZLIB:
-            vs->features |= VNC_FEATURE_ZLIB_MASK;
-            vs->vnc_encoding = enc;
+            /*
+             * VNC_ENCODING_ZRLE compresses better than VNC_ENCODING_ZLIB.
+             * So prioritize ZRLE, even if the client hints that it prefers
+             * ZLIB.
+             */
+            if ((vs->features & VNC_FEATURE_ZRLE_MASK) == 0) {
+                vs->features |= VNC_FEATURE_ZLIB_MASK;
+                vs->vnc_encoding = enc;
+            }
             break;
         case VNC_ENCODING_ZRLE:
             vs->features |= VNC_FEATURE_ZRLE_MASK;