Commit Graph

324 Commits

Author SHA1 Message Date
Markus Armbruster
552d7f49ee qom: Crash more nicely on object_property_get_link() failure
Pass &error_abort instead of NULL where the returned value is
dereferenced or asserted to be non-null.  Drop a now redundant
assertion.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200707160613.848843-24-armbru@redhat.com>
2020-07-10 15:18:08 +02:00
Markus Armbruster
b94b3c02df vnc: Plug minor memory leak in vnc_display_open()
vnc_display_print_local_addr() leaks the Error object when
qio_channel_socket_get_local_address() fails.  Seems unlikely.  Called
when we create a VNC display with vnc_display_open().  Plug the leak
by passing NULL to ignore the error.

Cc: Daniel P. Berrange <berrange@redhat.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200630090351.1247703-12-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-07-02 06:25:29 +02:00
Markus Armbruster
9261ef5e32 Clean up some calls to ignore Error objects the right way
Receiving the error in a local variable only to free it is less clear
(and also less efficient) than passing NULL.  Clean up.

Cc: Daniel P. Berrange <berrange@redhat.com>
Cc: Jerome Forissier <jerome@forissier.org>
CC: Greg Kurz <groug@kaod.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Greg Kurz <groug@kaod.org>
Message-Id: <20200630090351.1247703-4-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-07-02 06:25:28 +02:00
Philippe Mathieu-Daudé
57a878ed4f audio: Let capture_callback handler use const buffer argument
The buffer is the captured input to pass to backends.
As we should not modify it, mark the argument const.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200505132603.8575-3-f4bug@amsat.org>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2020-05-26 08:29:39 +02:00
Cameron Esfahani
557ba0e572 vnc: prioritize ZRLE compression over ZLIB
In my investigation, ZRLE always compresses better than ZLIB so
prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
preferred.

zlib buffer is always reset in zrle_compress_data(), so using offset to
calculate next_out and avail_out is useless.

Signed-off-by: Cameron Esfahani <dirty@apple.com>
Message-Id: <b5d129895d08a90d0a2a6183b95875bacfa998b8.1579582674.git.dirty@apple.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2020-01-21 07:29:03 +01:00
Gerd Hoffmann
0780ec7be8 Revert "vnc: allow fall back to RAW encoding"
This reverts commit de3f7de7f4.

Remove VNC optimization to reencode framebuffer update as raw if it's
smaller than the default encoding.

QEMU's implementation was naive and didn't account for the ZLIB z_stream
mutating with each compression.  Because of the mutation, simply
resetting the output buffer's offset wasn't sufficient to "rewind" the
operation.  The mutated z_stream would generate future zlib blocks which
referred to symbols in past blocks which weren't sent.  This would lead
to artifacting.

Considering that ZRLE is never larger than raw and even though ZLIB can
occasionally be fractionally larger than raw, the overhead of
implementing this optimization correctly isn't worth it.

Signed-off-by: Cameron Esfahani <dirty@apple.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2020-01-21 07:09:56 +01:00
Vladimir Sementsov-Ogievskiy
34ab29c2ff vnc: drop Error pointer indirection in vnc_client_io_error
We don't need Error **, as all callers pass local Error object, which
isn't used after the call, or NULL. Use Error * instead.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20191205174635.18758-6-vsementsov@virtuozzo.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2019-12-18 08:36:16 +01:00
Li Qiang
6bf21f3d83 vnc: fix memory leak when vnc disconnect
Currently when qemu receives a vnc connect, it creates a 'VncState' to
represent this connection. In 'vnc_worker_thread_loop' it creates a
local 'VncState'. The connection 'VcnState' and local 'VncState' exchange
data in 'vnc_async_encoding_start' and 'vnc_async_encoding_end'.
In 'zrle_compress_data' it calls 'deflateInit2' to allocate the libz library
opaque data. The 'VncState' used in 'zrle_compress_data' is the local
'VncState'. In 'vnc_zrle_clear' it calls 'deflateEnd' to free the libz
library opaque data. The 'VncState' used in 'vnc_zrle_clear' is the connection
'VncState'. In currently implementation there will be a memory leak when the
vnc disconnect. Following is the asan output backtrack:

Direct leak of 29760 byte(s) in 5 object(s) allocated from:
    0 0xffffa67ef3c3 in __interceptor_calloc (/lib64/libasan.so.4+0xd33c3)
    1 0xffffa65071cb in g_malloc0 (/lib64/libglib-2.0.so.0+0x571cb)
    2 0xffffa5e968f7 in deflateInit2_ (/lib64/libz.so.1+0x78f7)
    3 0xaaaacec58613 in zrle_compress_data ui/vnc-enc-zrle.c:87
    4 0xaaaacec58613 in zrle_send_framebuffer_update ui/vnc-enc-zrle.c:344
    5 0xaaaacec34e77 in vnc_send_framebuffer_update ui/vnc.c:919
    6 0xaaaacec5e023 in vnc_worker_thread_loop ui/vnc-jobs.c:271
    7 0xaaaacec5e5e7 in vnc_worker_thread ui/vnc-jobs.c:340
    8 0xaaaacee4d3c3 in qemu_thread_start util/qemu-thread-posix.c:502
    9 0xffffa544e8bb in start_thread (/lib64/libpthread.so.0+0x78bb)
    10 0xffffa53965cb in thread_start (/lib64/libc.so.6+0xd55cb)

This is because the opaque allocated in 'deflateInit2' is not freed in
'deflateEnd'. The reason is that the 'deflateEnd' calls 'deflateStateCheck'
and in the latter will check whether 's->strm != strm'(libz's data structure).
This check will be true so in 'deflateEnd' it just return 'Z_STREAM_ERROR' and
not free the data allocated in 'deflateInit2'.

The reason this happens is that the 'VncState' contains the whole 'VncZrle',
so when calling 'deflateInit2', the 's->strm' will be the local address.
So 's->strm != strm' will be true.

To fix this issue, we need to make 'zrle' of 'VncState' to be a pointer.
Then the connection 'VncState' and local 'VncState' exchange mechanism will
work as expection. The 'tight' of 'VncState' has the same issue, let's also turn
it to a pointer.

Reported-by: Ying Fang <fangying1@huawei.com>
Signed-off-by: Li Qiang <liq3ea@163.com>
Message-id: 20190831153922.121308-1-liq3ea@163.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-09-17 13:45:10 +02:00
Gerd Hoffmann
e1b3d47751 vnc: fix websocket field in events
Just need to fill VncClientInfo.websocket in vnc_client_cache_addr().

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1748175
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190904055250.22421-1-kraxel@redhat.com
2019-09-17 13:42:31 +02:00
Juan Quintela
fc8135c630 socket: Add num connections to qio_net_listener_open_sync()
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
2019-09-03 23:24:42 +02:00
Kővágó, Zoltán
f0b9f36d71 audio: add audiodev property to vnc and wav_capture
Signed-off-by: Kővágó, Zoltán <DirtY.iCE.hu@gmail.com>
Message-id: 8cbc9e865bbf40850c14340fc0549e6ac2d5fe9c.1566168923.git.DirtY.iCE.hu@gmail.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-08-21 09:13:37 +02:00
Kővágó, Zoltán
ecd97e9592 audio: basic support for multi backend audio
Audio functions no longer access glob_audio_state, instead they get an
AudioState as a parameter.  This is required in order to support
multiple backends.

glob_audio_state is also gone, and replaced with a tailq so we can store
more than one states.

Signed-off-by: Kővágó, Zoltán <DirtY.iCE.hu@gmail.com>
Message-id: 67aef54f9e729a7160fe95c465351115e392164b.1566168923.git.DirtY.iCE.hu@gmail.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-08-21 09:13:37 +02:00
Markus Armbruster
db72581598 Include qemu/main-loop.h less
In my "build everything" tree, changing qemu/main-loop.h triggers a
recompile of some 5600 out of 6600 objects (not counting tests and
objects that don't depend on qemu/osdep.h).  It includes block/aio.h,
which in turn includes qemu/event_notifier.h, qemu/notify.h,
qemu/processor.h, qemu/qsp.h, qemu/queue.h, qemu/thread-posix.h,
qemu/thread.h, qemu/timer.h, and a few more.

Include qemu/main-loop.h only where it's needed.  Touching it now
recompiles only some 1700 objects.  For block/aio.h and
qemu/event_notifier.h, these numbers drop from 5600 to 2800.  For the
others, they shrink only slightly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190812052359.30071-21-armbru@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-08-16 13:31:52 +02:00
Markus Armbruster
13d4ff07e8 trace: Do not include qom/cpu.h into generated trace.h
docs/devel/tracing.txt explains "since many source files include
trace.h, [the generated trace.h use] a minimum of types and other
header files included to keep the namespace clean and compile times
and dependencies down."

Commit 4815185902 "trace: Add per-vCPU tracing states for events with
the 'vcpu' property" made them all include qom/cpu.h via
control-internal.h.  qom/cpu.h in turn includes about thirty headers.
Ouch.

Per-vCPU tracing is currently not supported in sub-directories'
trace-events.  In other words, qom/cpu.h can only be used in
trace-root.h, not in any trace.h.

Split trace/control-vcpu.h off trace/control.h and
trace/control-internal.h.  Have the generated trace.h include
trace/control.h (which no longer includes qom/cpu.h), and trace-root.h
include trace/control-vcpu.h (which includes it).

The resulting improvement is a bit disappointing: in my "build
everything" tree, some 1100 out of 6600 objects (not counting tests
and objects that don't depend on qemu/osdep.h) depend on a trace.h,
and about 600 of them no longer depend on qom/cpu.h.  But more than
1300 others depend on trace-root.h.  More work is clearly needed.
Left for another day.

Cc: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20190812052359.30071-8-armbru@redhat.com>
2019-08-16 13:31:52 +02:00
Markus Armbruster
0b8fa32f55 Include qemu/module.h where needed, drop it from qemu-common.h
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190523143508.25387-4-armbru@redhat.com>
[Rebased with conflicts resolved automatically, except for
hw/usb/dev-hub.c hw/misc/exynos4210_rng.c hw/misc/bcm2835_rng.c
hw/misc/aspeed_scu.c hw/display/virtio-vga.c hw/arm/stm32f205_soc.c;
ui/cocoa.m fixed up]
2019-06-12 13:18:33 +02:00
Richard Henderson
f7b2502cdc ui/vnc: Use gcrypto_random_bytes for start_auth_vnc
Use a better interface for random numbers than rand().
Fail gracefully if for some reason we cannot use the crypto system.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2019-05-22 12:38:54 -04:00
Richard Henderson
4347e6383f ui/vnc: Split out authentication_failed
There were 3 copies of this code, one of which used the wrong
data size for the failure indicator.

Reviewed-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2019-05-22 12:38:54 -04:00
Peter Maydell
cfc3fef6b4 audio: introduce -audiodev
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.22 (GNU/Linux)
 
 iQIcBAABAgAGBQJch1vTAAoJEEy22O7T6HE4KL8QAI7VfMK8ZKTx/9Su46RL1A0G
 xNYhapdkhK2DEMd7JDY+eg8I8VQEwWroEjphCxLdx3p84Gr44gwgj3T24iRlXugq
 hJEgtBR548bBBdGd6KRihS8ZOro1T2gXol/TRq9z/TEF2LU6AVy52riZ/iydEd2U
 /ibkISR/vLdOnQ1Mj4YpchCZHx1yvMjSOhF/Cw2kXUm8C0jEuj3ws/BfdZQ8DxDG
 ayzS7JwSGOedsoFj+yfWX/YjCiocaXSLux9kyACzhHSJcA/5hw5srbXyhe4JrgdY
 BEqaUa23KiulgFt5fXnprktq+BQba/a4Tbx+YZFaNvX4HXqVccnCNN+VMkV4CvBN
 Y6UwijfaePVaQFP4kP+vpvqdPoJHNGikPlX8j3Q7ofL5+/c7Qk9yChNyB8lpfOIe
 KBNNsIIJO2GVU0IVNuqIGldWZYQw2y8ojpNSntg5lyFIrwZ+ipDFuxD9weZibVgc
 pa4VfPNKW4lOWrDX0PLv5eTNWOLamY3T1wY6pQCOtHgzlChRa28pMWkB7VkzTTQW
 gzipigqpVzmd3l1m2uI7LOovZa326IrGRC1/Yb4a41Gz58p9a1U2sYTijhPlAaeq
 VDMafPTAIJYpg0MW5Uxh1eB05WzNafPfEznjMJ/Z84J4P/oGFxlOGcL99sAbrxpl
 cQWWZlBGINQhxZ1PtvJc
 =7YpW
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kraxel/tags/audio-20190312-pull-request' into staging

audio: introduce -audiodev

# gpg: Signature made Tue 12 Mar 2019 07:12:19 GMT
# gpg:                using RSA key 4CB6D8EED3E87138
# gpg: Good signature from "Gerd Hoffmann (work) <kraxel@redhat.com>" [full]
# gpg:                 aka "Gerd Hoffmann <gerd@kraxel.org>" [full]
# gpg:                 aka "Gerd Hoffmann (private) <kraxel@gmail.com>" [full]
# Primary key fingerprint: A032 8CFF B93A 17A7 9901  FE7D 4CB6 D8EE D3E8 7138

* remotes/kraxel/tags/audio-20190312-pull-request:
  audio: -audiodev command line option: cleanup
  wavaudio: port to -audiodev config
  spiceaudio: port to -audiodev config
  sdlaudio: port to -audiodev config
  paaudio: port to -audiodev config
  ossaudio: port to -audiodev config
  noaudio: port to -audiodev config
  dsoundaudio: port to -audiodev config
  coreaudio: port to -audiodev config
  alsaaudio: port to -audiodev config
  audio: -audiodev command line option basic implementation
  audio: -audiodev command line option: documentation
  audio: use qapi AudioFormat instead of audfmt_e
  qapi: qapi for audio backends

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

# Conflicts:
#	qemu-deprecated.texi
2019-03-12 16:45:13 +00:00
Kővágó, Zoltán
85bc58520c audio: use qapi AudioFormat instead of audfmt_e
I had to include an enum for audio sampling formats into qapi, but that
meant duplicating the audfmt_e enum.  This patch replaces audfmt_e and
associated values with the qapi generated AudioFormat enum.

This patch is mostly a search-and-replace, except for switches where the
qapi generated AUDIO_FORMAT_MAX caused problems.

Signed-off-by: Kővágó, Zoltán <DirtY.iCE.hu@gmail.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-id: 01251b2758a1679c66842120b77c0fb46d7d0eaf.1552083282.git.DirtY.iCE.hu@gmail.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-03-11 10:29:26 +01:00
Daniel P. Berrange
55cf09a02b vnc: allow specifying a custom authorization object name
The VNC server has historically had support for ACLs to check both the
SASL username and the TLS x509 distinguished name. The VNC server was
responsible for creating the initial ACL, and the client app was then
responsible for populating it with rules using the HMP 'acl_add' command.

This is not satisfactory for a variety of reasons. There is no way to
populate the ACLs from the command line, users are forced to use the
HMP. With multiple network services all supporting TLS and ACLs now, it
is desirable to be able to define a single ACL that is referenced by all
services.

To address these limitations, two new options are added to the VNC
server CLI. The 'tls-authz' option takes the ID of a QAuthZ object to
use for checking TLS x509 distinguished names, and the 'sasl-authz'
option takes the ID of another object to use for checking SASL usernames.

In this example, we setup two authorization rules. The first allows any
client with a certificate issued by the 'RedHat' organization in the
'London' locality. The second ACL allows clients with either the
'joe@REDHAT.COM' or  'fred@REDHAT.COM' kerberos usernames. Both checks
must pass for the user to be allowed.

    $QEMU -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
                  endpoint=server,verify-peer=yes \
          -object authz-simple,id=authz0,policy=deny,\
                  rules.0.match=O=RedHat,,L=London,rules.0.policy=allow \
          -object authz-simple,id=authz1,policy=deny,\
                  rules.0.match=fred@REDHAT.COM,rules.0.policy=allow \
                  rules.0.match=joe@REDHAT.COM,rules.0.policy=allow \
          -vnc 0.0.0.0:1,tls-creds=tls0,tls-authz=authz0,
	       sasl,sasl-authz=authz1 \
          ...other QEMU args...

Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 20190227145755.26556-2-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-03-11 08:39:02 +01:00
Gerd Hoffmann
bbcdeb623d vnc: fix update stalls
vnc aborts display update jobs on video mode switches and page flips.
That can cause vnc update stalls in case an unfinished vnc job gets
aborted.  The vnc client will never receive the requested update then.
Fix that by copying the state from job_update back to update in that
case.

Reports complain about stalls with two or more clients being connected
at the same time, on some but not all connections.  I suspect it can
also happen with a single connection, multiple connections only make
this more much likely to happen.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1662260
Reported-by: Ying Fang <fangying1@huawei.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Ying Fang <fangying1@huawei.com>
Message-id: 20190305130930.24516-1-kraxel@redhat.com
2019-03-11 08:39:02 +01:00
Daniel P. Berrange
b76806d4ec authz: delete existing ACL implementation
The 'qemu_acl' type was a previous non-QOM based attempt to provide an
authorization facility in QEMU. Because it is non-QOM based it cannot be
created via the command line and requires special monitor commands to
manipulate it.

The new QAuthZ subclasses provide a superset of the functionality in
qemu_acl, so the latter can now be deleted. The HMP 'acl_*' monitor
commands are converted to use the new QAuthZSimple data type instead
in order to provide temporary backwards compatibility.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2019-02-26 15:32:19 +00:00
Markus Armbruster
5d75648b56 qapi: Generate QAPIEvent stuff into separate files
Having to include qapi-events.h just for QAPIEvent is suboptimal, but
quite tolerable now.  It'll become problematic when we have events
conditional on the target, because then qapi-events.h won't be usable
from target-independent code anymore.  Avoid that by generating it
into separate files.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190214152251.2073-6-armbru@redhat.com>
2019-02-18 14:44:04 +01:00
Gerd Hoffmann
19c1b9fd3d keymap: fix keyup mappings
It is possible that the modifier state on keyup is different from the
modifier state on keydown.  In that case the keycode lookup can end up
with different keys in case multiple keysym -> keycode mappings exist,
because it picks the mapping depending on modifier state.

To fix that change the lookup logic for keyup events.  Instead of
looking at the modifier state check the key state and prefer a keycodes
where the key is in "down" state right now.

Fixes: abb4f2c965 keymap: consider modifier state when picking a mapping
Buglink: https://bugs.launchpad.net/bugs/1738283
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1658676
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20190122092814.14919-9-kraxel@redhat.com
2019-02-05 10:45:44 +01:00
Gerd Hoffmann
4ed26e19d9 keymap: pass full keyboard state to keysym2scancode
Pass the keyboard state tracker handle down to keysym2scancode(),
so the code can fully inspect the keyboard state as needed.  No
functional change.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20190122092814.14919-8-kraxel@redhat.com
2019-02-05 10:45:44 +01:00
Gerd Hoffmann
c2f2ba4983 kbd-state: use state tracker for vnc
Use the new keyboard state tracked for vnc.  Allows to drop the
vnc-specific modifier state tracking code.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20190122092814.14919-7-kraxel@redhat.com
2019-02-05 10:45:44 +01:00
Gerd Hoffmann
61e77a5f0c vnc: detect and optimize pageflips
When size and format of the display surface stays the same we can just
tag the guest display as dirty and be done with it.

There is no need need to resize the vnc server display or to touch the
vnc client dirty bits.  On the next refresh cycle
vnc_refresh_server_surface() will check for actual display content
changes and update the client dirty bits as needed.

The desktop resize and framebuffer format notifications to the vnc
client will be skipped too.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20190116101049.8929-1-kraxel@redhat.com
2019-01-21 09:43:13 +01:00
Paolo Bonzini
7d37435bd5 avoid TABs in files that only contain a few
Most files that have TABs only contain a handful of them.  Change
them to spaces so that we don't confuse people.

disas, standard-headers, linux-headers and libdecnumber are imported
from other projects and probably should be exempted from the check.
Outside those, after this patch the following files still contain both
8-space and TAB sequences at the beginning of the line.  Many of them
have a majority of TABs, or were initially committed with all tabs.

    bsd-user/i386/target_syscall.h
    bsd-user/x86_64/target_syscall.h
    crypto/aes.c
    hw/audio/fmopl.c
    hw/audio/fmopl.h
    hw/block/tc58128.c
    hw/display/cirrus_vga.c
    hw/display/xenfb.c
    hw/dma/etraxfs_dma.c
    hw/intc/sh_intc.c
    hw/misc/mst_fpga.c
    hw/net/pcnet.c
    hw/sh4/sh7750.c
    hw/timer/m48t59.c
    hw/timer/sh_timer.c
    include/crypto/aes.h
    include/disas/bfd.h
    include/hw/sh4/sh.h
    libdecnumber/decNumber.c
    linux-headers/asm-generic/unistd.h
    linux-headers/linux/kvm.h
    linux-user/alpha/target_syscall.h
    linux-user/arm/nwfpe/double_cpdo.c
    linux-user/arm/nwfpe/fpa11_cpdt.c
    linux-user/arm/nwfpe/fpa11_cprt.c
    linux-user/arm/nwfpe/fpa11.h
    linux-user/flat.h
    linux-user/flatload.c
    linux-user/i386/target_syscall.h
    linux-user/ppc/target_syscall.h
    linux-user/sparc/target_syscall.h
    linux-user/syscall.c
    linux-user/syscall_defs.h
    linux-user/x86_64/target_syscall.h
    slirp/cksum.c
    slirp/if.c
    slirp/ip.h
    slirp/ip_icmp.c
    slirp/ip_icmp.h
    slirp/ip_input.c
    slirp/ip_output.c
    slirp/mbuf.c
    slirp/misc.c
    slirp/sbuf.c
    slirp/socket.c
    slirp/socket.h
    slirp/tcp_input.c
    slirp/tcpip.h
    slirp/tcp_output.c
    slirp/tcp_subr.c
    slirp/tcp_timer.c
    slirp/tftp.c
    slirp/udp.c
    slirp/udp.h
    target/cris/cpu.h
    target/cris/mmu.c
    target/cris/op_helper.c
    target/sh4/helper.c
    target/sh4/op_helper.c
    target/sh4/translate.c
    tcg/sparc/tcg-target.inc.c
    tests/tcg/cris/check_addo.c
    tests/tcg/cris/check_moveq.c
    tests/tcg/cris/check_swap.c
    tests/tcg/multiarch/test-mmap.c
    ui/vnc-enc-hextile-template.h
    ui/vnc-enc-zywrle.h
    util/envlist.c
    util/readline.c

The following have only TABs:

    bsd-user/i386/target_signal.h
    bsd-user/sparc64/target_signal.h
    bsd-user/sparc64/target_syscall.h
    bsd-user/sparc/target_signal.h
    bsd-user/sparc/target_syscall.h
    bsd-user/x86_64/target_signal.h
    crypto/desrfb.c
    hw/audio/intel-hda-defs.h
    hw/core/uboot_image.h
    hw/sh4/sh7750_regnames.c
    hw/sh4/sh7750_regs.h
    include/hw/cris/etraxfs_dma.h
    linux-user/alpha/termbits.h
    linux-user/arm/nwfpe/fpopcode.h
    linux-user/arm/nwfpe/fpsr.h
    linux-user/arm/syscall_nr.h
    linux-user/arm/target_signal.h
    linux-user/cris/target_signal.h
    linux-user/i386/target_signal.h
    linux-user/linux_loop.h
    linux-user/m68k/target_signal.h
    linux-user/microblaze/target_signal.h
    linux-user/mips64/target_signal.h
    linux-user/mips/target_signal.h
    linux-user/mips/target_syscall.h
    linux-user/mips/termbits.h
    linux-user/ppc/target_signal.h
    linux-user/sh4/target_signal.h
    linux-user/sh4/termbits.h
    linux-user/sparc64/target_syscall.h
    linux-user/sparc/target_signal.h
    linux-user/x86_64/target_signal.h
    linux-user/x86_64/termbits.h
    pc-bios/optionrom/optionrom.h
    slirp/mbuf.h
    slirp/misc.h
    slirp/sbuf.h
    slirp/tcp.h
    slirp/tcp_timer.h
    slirp/tcp_var.h
    target/i386/svm.h
    target/sparc/asi.h
    target/xtensa/core-dc232b/xtensa-modules.inc.c
    target/xtensa/core-dc233c/xtensa-modules.inc.c
    target/xtensa/core-de212/core-isa.h
    target/xtensa/core-de212/xtensa-modules.inc.c
    target/xtensa/core-fsf/xtensa-modules.inc.c
    target/xtensa/core-sample_controller/core-isa.h
    target/xtensa/core-sample_controller/xtensa-modules.inc.c
    target/xtensa/core-test_kc705_be/core-isa.h
    target/xtensa/core-test_kc705_be/xtensa-modules.inc.c
    tests/tcg/cris/check_abs.c
    tests/tcg/cris/check_addc.c
    tests/tcg/cris/check_addcm.c
    tests/tcg/cris/check_addoq.c
    tests/tcg/cris/check_bound.c
    tests/tcg/cris/check_ftag.c
    tests/tcg/cris/check_int64.c
    tests/tcg/cris/check_lz.c
    tests/tcg/cris/check_openpf5.c
    tests/tcg/cris/check_sigalrm.c
    tests/tcg/cris/crisutils.h
    tests/tcg/cris/sys.c
    tests/tcg/i386/test-i386-ssse3.c
    ui/vgafont.h

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20181213223737.11793-3-pbonzini@redhat.com>
Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Eric Blake <eblake@redhat.com>
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Reviewed-by: Stefan Markovic <smarkovic@wavecomp.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-01-11 15:46:56 +01:00
Markus Armbruster
612aea2017 vnc: Clean up error reporting in vnc_init_func()
Calling error_report() in a function that takes an Error ** argument
is suspicious.  vnc_init_func() does that, and then fails without
setting an error.  Its caller main(), via qemu_opts_foreach(), is fine
with it, but clean it up anyway.

While there, drop a "Failed to start VNC server: " error message
prefix that doesn't really add value.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20181017082702.5581-28-armbru@redhat.com>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
2018-10-19 14:51:34 +02:00
Fei Li
ab4f931e9f ui: Convert vnc_display_init(), init_keyboard_layout() to Error
Signed-off-by: Fei Li <fli@suse.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20181017082702.5581-27-armbru@redhat.com>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
2018-10-19 14:51:34 +02:00
Marc-André Lureau
b5dc0d7d56 vnc: call sasl_server_init() only when required
VNC server is calling sasl_server_init() during startup of QEMU, even
if SASL auth has not been enabled.

This may create undesirable warnings like "Could not find keytab file:
/etc/qemu/krb5.tab" when the user didn't configure SASL on host and
started VNC server.

Instead, only initialize SASL when needed. Note that HMP/QMP "change
vnc" calls vnc_display_open() again, which will initialize SASL if
needed.

Fix assignment in if condition, while touching this code.

Related to:
https://bugzilla.redhat.com/show_bug.cgi?id=1609327

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20180907063634.359-1-marcandre.lureau@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-10-01 11:29:03 +02:00
Peter Xu
3ab72385b2 qapi: Drop qapi_event_send_FOO()'s Error ** argument
The generated qapi_event_send_FOO() take an Error ** argument.  They
can't actually fail, because all they do with the argument is passing it
to functions that can't fail: the QObject output visitor, and the
@qmp_emit callback, which is either monitor_qapi_event_queue() or
event_test_emit().

Drop the argument, and pass &error_abort to the QObject output visitor
and @qmp_emit instead.

Suggested-by: Eric Blake <eblake@redhat.com>
Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180815133747.25032-4-peterx@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten, update to qapi-code-gen.txt corrected]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2018-08-28 18:21:38 +02:00
Philippe Mathieu-Daudé
949ed4c227 ui/vnc: Remove useless parenthesis around DIV_ROUND_UP macro
Patch created mechanically by rerunning:

  $  spatch --sp-file scripts/coccinelle/round.cocci \
            --macro-file scripts/cocci-macro-file.h \
            --dir . --in-place

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20180704153919.12432-7-f4bug@amsat.org>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-08-24 08:40:11 +02:00
Daniel P. Berrangé
ec86faa934 vnc: remove support for deprecated tls, x509, x509verify options
The 'tls-creds' option accepts the name of a TLS credentials
object. This replaced the usage of 'tls', 'x509' and 'x509verify'
options in 2.5.0. These deprecated options were grandfathered in
when the deprecation policy was introduded in 2.10.0, so can now
finally be removed.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20180725092751.21767-3-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-08-24 08:40:11 +02:00
Gerd Hoffmann
f31f9c1080 vnc: add magic cookie to VncState
Set magic cookie on initialization.  Clear on cleanup.  Sprinkle a bunch
of assert()s checking the cookie, to verify the pointer is valid.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Message-id: 20180507102254.12107-1-kraxel@redhat.com
2018-05-15 10:36:55 +02:00
Gerd Hoffmann
1bc3117aba vnc: fix use-after-free
When vnc_client_read() return value is -1
vs is not valid any more.

Fixes: d49b87f0d1
Reported-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-id: 20180420084820.3873-1-kraxel@redhat.com
2018-04-27 09:51:23 +02:00
Gerd Hoffmann
2e5567c903 vnc: deal with surface NULL pointers
Secondary displays in multihead setups are allowed to have a NULL
DisplaySurface.  Typically user interfaces handle this by hiding the
window which shows the display in question.

This isn't an option for vnc though because it simply hasn't a concept
of windows or outputs.  So handle the situation by showing a placeholder
DisplaySurface instead.  Also check in console_select whenever a surface
is preset in the first place before requesting an update.

This fixes a segfault which can be triggered by switching to an unused
display (via vtrl-alt-<nr>) in a multihead setup, for example using
-device virtio-vga,max_outputs=2.

Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
Message-id: 20180308161803.6152-1-kraxel@redhat.com
2018-03-12 09:00:59 +01:00
Markus Armbruster
eb815e248f qapi: Move qapi-schema.json to qapi/, rename generated files
Move qapi-schema.json to qapi/, so it's next to its modules, and all
files get generated to qapi/, not just the ones generated for modules.

Consistently name the generated files qapi-MODULE.EXT:
qmp-commands.[ch] become qapi-commands.[ch], qapi-event.[ch] become
qapi-events.[ch], and qmp-introspect.[ch] become qapi-introspect.[ch].
This gets rid of the temporary hacks in scripts/qapi/commands.py,
scripts/qapi/events.py, and scripts/qapi/common.py.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-28-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: Fix trailing dot in tpm.c, undo temporary hack for OSX toolchain]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:45:57 -06:00
Markus Armbruster
9af2398977 Include less of the generated modular QAPI headers
In my "build everything" tree, a change to the types in
qapi-schema.json triggers a recompile of about 4800 out of 5100
objects.

The previous commit split up qmp-commands.h, qmp-event.h, qmp-visit.h,
qapi-types.h.  Each of these headers still includes all its shards.
Reduce compile time by including just the shards we actually need.

To illustrate the benefits: adding a type to qapi/migration.json now
recompiles some 2300 instead of 4800 objects.  The next commit will
improve it further.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-24-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
[eblake: rebase to master]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:45:50 -06:00
Gerd Hoffmann
abb4f2c965 keymap: consider modifier state when picking a mapping
Pass the modifier state to the keymap lookup function.  In case multiple
keysym -> keycode mappings exist look at the modifier state and prefer
the mapping where the modifier state matches.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20180222070513.8740-6-kraxel@redhat.com
2018-02-22 10:35:32 +01:00
Daniel P. Berrangé
cf0706581b ui: check VNC audio frequency limit at time of reading from client
The 'vs->as.freq' value is a signed integer, which is read from an
unsigned 32-bit int field on the wire. There is thus a risk of overflow
on 32-bit platforms. Move the frequency limit checking to be done at
time of read before casting to a signed integer.

Reported-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-id: 20180205114938.15784-4-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-02-16 12:33:02 +01:00
Daniel P. Berrangé
dffa1de071 ui: avoid risk of 32-bit int overflow in VNC buffer check
For very large framebuffers, it is theoretically possible for the result
of 'vs->throttle_output_offset * VNC_THROTTLE_OUTPUT_LIMIT_SCALE' to
exceed the size of a 32-bit int. For this to happen in practice, the
video RAM would have to be set to a large enough value, which is not
likely today. None the less we can be paranoid against future growth by
using division instead of multiplication when checking the limits.

Reported-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-id: 20180205114938.15784-2-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-02-16 12:33:02 +01:00
Klim Kireev
d49b87f0d1 vnc: fix segfault in closed connection handling
On one of our client's node, due to trying to read from closed ioc,
a segmentation fault occured. Corresponding backtrace:

0  object_get_class (obj=obj@entry=0x0)
1  qio_channel_readv_full (ioc=0x0, iov=0x7ffe55277180 ...
2  qio_channel_read (ioc=<optimized out> ...
3  vnc_client_read_buf (vs=vs@entry=0x55625f3c6000, ...
4  vnc_client_read_plain (vs=0x55625f3c6000)
5  vnc_client_read (vs=0x55625f3c6000)
6  vnc_client_io (ioc=<optimized out>, condition=G_IO_IN, ...
7  g_main_dispatch (context=0x556251568a50)
8  g_main_context_dispatch (context=context@entry=0x556251568a50)
9  glib_pollfds_poll ()
10 os_host_main_loop_wait (timeout=<optimized out>)
11 main_loop_wait (nonblocking=nonblocking@entry=0)
12 main_loop () at vl.c:1909
13 main (argc=<optimized out>, argv=<optimized out>, ...

Having analyzed the coredump, I understood that the reason is that
ioc_tag is reset on vnc_disconnect_start and ioc is cleaned
in vnc_disconnect_finish. Between these two events due to some
reasons the ioc_tag was set again and after vnc_disconnect_finish
the handler is running with freed ioc,
which led to the segmentation fault.

The patch checks vs->disconnecting in places where we call
qio_channel_add_watch and resets handler if disconnecting == TRUE
to prevent such an occurrence.

Signed-off-by: Klim Kireev <klim.kireev@virtuozzo.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20180207094844.21402-1-klim.kireev@virtuozzo.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-02-16 12:23:54 +01:00
Daniel P. Berrangé
f42fdb24b7 vnc: remove bogus object_unref on client socket
vnc_listen_io() does not own the reference on the 'cioc' parameter is it
passed, so should not be unref'ing it.

Fixes: 13e1d0e71e
Reported-by: Bandan Das <bsd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20180215102602.10864-1-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-02-16 12:22:35 +01:00
Markus Armbruster
922a01a013 Move include qemu/option.h from qemu-common.h to actual users
qemu-common.h includes qemu/option.h, but most places that include the
former don't actually need the latter.  Drop the include, and add it
to the places that actually need it.

While there, drop superfluous includes of both headers, and
separate #include from file comment with a blank line.

This cleanup makes the number of objects depending on qemu/option.h
drop from 4545 (out of 4743) to 284 in my "build everything" tree.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-20-armbru@redhat.com>
[Semantic conflict with commit bdd6a90a9e in block/nvme.c resolved]
2018-02-09 13:52:16 +01:00
Markus Armbruster
6b67395762 Eliminate qapi/qmp/types.h
qapi/qmp/types.h is a convenience header to include a number of
qapi/qmp/ headers.  Since we rarely need all of the headers
qapi/qmp/types.h includes, we bypass it most of the time.  Most of the
places that use it don't need all the headers, either.

Include the necessary headers directly, and drop qapi/qmp/types.h.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-9-armbru@redhat.com>
2018-02-09 13:52:15 +01:00
Markus Armbruster
a82400cf5c Drop superfluous includes of qapi/qmp/qerror.h
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-6-armbru@redhat.com>
2018-02-09 13:51:35 +01:00
Markus Armbruster
e688df6bc4 Include qapi/error.h exactly where needed
This cleanup makes the number of objects depending on qapi/error.h
drop from 1910 (out of 4743) to 1612 in my "build everything" tree.

While there, separate #include from file comment with a blank line,
and drop a useless comment on why qemu/osdep.h is included first.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-5-armbru@redhat.com>
[Semantic conflict with commit 34e304e975 resolved, OSX breakage fixed]
2018-02-09 13:50:17 +01:00
Daniel P. Berrange
13e1d0e71e ui: convert VNC server to QIONetListener
The VNC server already has the ability to listen on multiple sockets.
Converting it to use the QIONetListener APIs though, will reduce the
amount of code in the VNC server and improve the clarity of what is
left.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 20180201164514.10330-1-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-02-02 07:47:39 +01:00
Daniel P. Berrange
4c956bd81e ui: avoid sign extension using client width/height
Pixman returns a signed int for the image width/height, but the VNC
protocol only permits a unsigned int16. Effective framebuffer size
is determined by the guest, limited by the video RAM size, so the
dimensions are unlikely to exceed the range of an unsigned int16,
but this is not currently validated.

With the current use of 'int' for client width/height, the calculation
of offsets in vnc_update_throttle_offset() suffers from integer size
promotion and sign extension, causing coverity warnings

*** CID 1385147:  Integer handling issues  (SIGN_EXTENSION)
/ui/vnc.c: 979 in vnc_update_throttle_offset()
973      * than that the client would already suffering awful audio
974      * glitches, so dropping samples is no worse really).
975      */
976     static void vnc_update_throttle_offset(VncState *vs)
977     {
978         size_t offset =
>>>     CID 1385147:  Integer handling issues  (SIGN_EXTENSION)
>>>     Suspicious implicit sign extension:
    "vs->client_pf.bytes_per_pixel" with type "unsigned char" (8 bits,
    unsigned) is promoted in "vs->client_width * vs->client_height *
    vs->client_pf.bytes_per_pixel" to type "int" (32 bits, signed), then
    sign-extended to type "unsigned long" (64 bits, unsigned).  If
    "vs->client_width * vs->client_height * vs->client_pf.bytes_per_pixel"
    is greater than 0x7FFFFFFF, the upper bits of the result will all be 1.
979             vs->client_width * vs->client_height * vs->client_pf.bytes_per_pixel;

Change client_width / client_height to be a size_t to avoid sign
extension and integer promotion. Then validate that dimensions are in
range wrt the RFB protocol u16 limits.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 20180118155254.17053-1-berrange@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-01-25 15:02:00 +01:00