From 1b5c15cebd5cc7595337a8a7cf14f6a707605e99 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Sat, 7 Sep 2019 12:20:55 -0500 Subject: [PATCH 1/5] nbd/client: Add hint when TLS is missing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I received an off-list report of failure to connect to an NBD server expecting an x509 certificate, when the client was attempting something similar to this command line: $ ./x86_64-softmmu/qemu-system-x86_64 -name 'blah' -machine q35 -nodefaults \ -object tls-creds-x509,id=tls0,endpoint=client,dir=$path_to_certs \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x6 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=nbd:localhost:9000,werror=stop,rerror=stop,tls-creds=tls0 \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 qemu-system-x86_64: -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=nbd:localhost:9000,werror=stop,rerror=stop,tls-creds=tls0: TLS negotiation required before option 7 (go) server reported: Option 0x7 not permitted before TLS The problem? As specified, -drive is trying to pass tls-creds to the raw format driver instead of the nbd protocol driver, but before we get to the point where we can detect that raw doesn't know what to do with tls-creds, the nbd driver has already failed because the server complained. The fix to the broken command line? Pass '...,file.tls-creds=tls0' to ensure the tls-creds option is handed to nbd, not raw. But since the error message was rather cryptic, I'm trying to improve the error message. With this patch, the error message adds a line: qemu-system-x86_64: -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=nbd:localhost:9000,werror=stop,rerror=stop,tls-creds=tls0: TLS negotiation required before option 7 (go) Did you forget a valid tls-creds? server reported: Option 0x7 not permitted before TLS And with luck, someone grepping for that error message will find this commit message and figure out their command line mistake. Sadly, the only mention of file.tls-creds in our docs relates to an --image-opts use of PSK encryption with qemu-img as the client, rather than x509 certificate encryption with qemu-kvm as the client. CC: Tingting Mao CC: Daniel P. Berrangé Signed-off-by: Eric Blake Message-Id: <20190907172055.26870-1-eblake@redhat.com> [eblake: squash in iotest 233 fix] Reviewed-by: Daniel P. Berrangé --- nbd/client.c | 1 + tests/qemu-iotests/233.out | 2 ++ 2 files changed, 3 insertions(+) diff --git a/nbd/client.c b/nbd/client.c index b9dc829175..f6733962b4 100644 --- a/nbd/client.c +++ b/nbd/client.c @@ -204,6 +204,7 @@ static int nbd_handle_reply_err(QIOChannel *ioc, NBDOptionReply *reply, case NBD_REP_ERR_TLS_REQD: error_setg(errp, "TLS negotiation required before option %" PRIu32 " (%s)", reply->option, nbd_opt_lookup(reply->option)); + error_append_hint(errp, "Did you forget a valid tls-creds?\n"); break; case NBD_REP_ERR_UNKNOWN: diff --git a/tests/qemu-iotests/233.out b/tests/qemu-iotests/233.out index 24321efa11..c3c344811b 100644 --- a/tests/qemu-iotests/233.out +++ b/tests/qemu-iotests/233.out @@ -21,8 +21,10 @@ server reported: TLS not configured == check plain client to TLS server fails == qemu-img: Could not open 'nbd://localhost:PORT': TLS negotiation required before option 7 (go) +Did you forget a valid tls-creds? server reported: Option 0x7 not permitted before TLS qemu-nbd: TLS negotiation required before option 3 (list) +Did you forget a valid tls-creds? server reported: Option 0x3 not permitted before TLS == check TLS works == From b4961249af0403fa55aae57c4c8806b24f7a7b33 Mon Sep 17 00:00:00 2001 From: Sergio Lopez Date: Thu, 12 Sep 2019 13:00:33 +0200 Subject: [PATCH 2/5] nbd/server: attach client channel to the export's AioContext On creation, the export's AioContext is set to the same one as the BlockBackend, while the AioContext in the client QIOChannel is left untouched. As a result, when using data-plane, nbd_client_receive_next_request() schedules coroutines in the IOThread AioContext, while the client's QIOChannel is serviced from the main_loop, potentially triggering the assertion at qio_channel_restart_[read|write]. To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached(). RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1748253 Signed-off-by: Sergio Lopez Message-Id: <20190912110032.26395-1-slp@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- nbd/server.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/nbd/server.c b/nbd/server.c index 28c3c8be85..31d624e146 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1297,6 +1297,11 @@ static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp) return ret; } + /* Attach the channel to the same AioContext as the export */ + if (client->exp && client->exp->ctx) { + qio_channel_attach_aio_context(client->ioc, client->exp->ctx); + } + assert(!client->optlen); trace_nbd_negotiate_success(); From 61bc846d8c58535af6884b637a4005dd6111ea95 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Mon, 16 Sep 2019 21:39:17 -0500 Subject: [PATCH 3/5] nbd: Grab aio context lock in more places When iothreads are in use, the failure to grab the aio context results in an assertion failure when trying to unlock things during blk_unref, when trying to unlock a mutex that was not locked. In short, all calls to nbd_export_put need to done while within the correct aio context. But since nbd_export_put can recursively reach itself via nbd_export_close, and recursively grabbing the context would deadlock, we can't do the context grab directly in those functions, but must do so in their callers. Hoist the use of the correct aio_context from nbd_export_new() to its caller qmp_nbd_server_add(). Then tweak qmp_nbd_server_remove(), nbd_eject_notifier(), and nbd_esport_close_all() to grab the right context, so that all callers during qemu now own the context before nbd_export_put() can call blk_unref(). Remaining uses in qemu-nbd don't matter (since that use case does not support iothreads). Suggested-by: Kevin Wolf Signed-off-by: Eric Blake Message-Id: <20190917023917.32226-1-eblake@redhat.com> Reviewed-by: Sergio Lopez --- blockdev-nbd.c | 14 ++++++++++++-- include/block/nbd.h | 1 + nbd/server.c | 22 ++++++++++++++++++---- 3 files changed, 31 insertions(+), 6 deletions(-) diff --git a/blockdev-nbd.c b/blockdev-nbd.c index 213f226ac1..6a8b206e1d 100644 --- a/blockdev-nbd.c +++ b/blockdev-nbd.c @@ -151,6 +151,7 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name, BlockBackend *on_eject_blk; NBDExport *exp; int64_t len; + AioContext *aio_context; if (!nbd_server) { error_setg(errp, "NBD server not running"); @@ -173,11 +174,13 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name, return; } + aio_context = bdrv_get_aio_context(bs); + aio_context_acquire(aio_context); len = bdrv_getlength(bs); if (len < 0) { error_setg_errno(errp, -len, "Failed to determine the NBD export's length"); - return; + goto out; } if (!has_writable) { @@ -190,13 +193,16 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name, exp = nbd_export_new(bs, 0, len, name, NULL, bitmap, !writable, !writable, NULL, false, on_eject_blk, errp); if (!exp) { - return; + goto out; } /* The list of named exports has a strong reference to this export now and * our only way of accessing it is through nbd_export_find(), so we can drop * the strong reference that is @exp. */ nbd_export_put(exp); + + out: + aio_context_release(aio_context); } void qmp_nbd_server_remove(const char *name, @@ -204,6 +210,7 @@ void qmp_nbd_server_remove(const char *name, Error **errp) { NBDExport *exp; + AioContext *aio_context; if (!nbd_server) { error_setg(errp, "NBD server not running"); @@ -220,7 +227,10 @@ void qmp_nbd_server_remove(const char *name, mode = NBD_SERVER_REMOVE_MODE_SAFE; } + aio_context = nbd_export_aio_context(exp); + aio_context_acquire(aio_context); nbd_export_remove(exp, mode, errp); + aio_context_release(aio_context); } void qmp_nbd_server_stop(Error **errp) diff --git a/include/block/nbd.h b/include/block/nbd.h index 21550747cf..316fd705a9 100644 --- a/include/block/nbd.h +++ b/include/block/nbd.h @@ -340,6 +340,7 @@ void nbd_export_put(NBDExport *exp); BlockBackend *nbd_export_get_blockdev(NBDExport *exp); +AioContext *nbd_export_aio_context(NBDExport *exp); NBDExport *nbd_export_find(const char *name); void nbd_export_close_all(void); diff --git a/nbd/server.c b/nbd/server.c index 31d624e146..d8d1e62455 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1461,7 +1461,12 @@ static void blk_aio_detach(void *opaque) static void nbd_eject_notifier(Notifier *n, void *data) { NBDExport *exp = container_of(n, NBDExport, eject_notifier); + AioContext *aio_context; + + aio_context = exp->ctx; + aio_context_acquire(aio_context); nbd_export_close(exp); + aio_context_release(aio_context); } NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset, @@ -1480,12 +1485,11 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset, * NBD exports are used for non-shared storage migration. Make sure * that BDRV_O_INACTIVE is cleared and the image is ready for write * access since the export could be available before migration handover. + * ctx was acquired in the caller. */ assert(name); ctx = bdrv_get_aio_context(bs); - aio_context_acquire(ctx); bdrv_invalidate_cache(bs, NULL); - aio_context_release(ctx); /* Don't allow resize while the NBD server is running, otherwise we don't * care what happens with the node. */ @@ -1493,7 +1497,7 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset, if (!readonly) { perm |= BLK_PERM_WRITE; } - blk = blk_new(bdrv_get_aio_context(bs), perm, + blk = blk_new(ctx, perm, BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE | BLK_PERM_GRAPH_MOD); ret = blk_insert_bs(blk, bs, errp); @@ -1560,7 +1564,7 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset, } exp->close = close; - exp->ctx = blk_get_aio_context(blk); + exp->ctx = ctx; blk_add_aio_context_notifier(blk, blk_aio_attached, blk_aio_detach, exp); if (on_eject_blk) { @@ -1593,6 +1597,12 @@ NBDExport *nbd_export_find(const char *name) return NULL; } +AioContext * +nbd_export_aio_context(NBDExport *exp) +{ + return exp->ctx; +} + void nbd_export_close(NBDExport *exp) { NBDClient *client, *next; @@ -1687,9 +1697,13 @@ BlockBackend *nbd_export_get_blockdev(NBDExport *exp) void nbd_export_close_all(void) { NBDExport *exp, *next; + AioContext *aio_context; QTAILQ_FOREACH_SAFE(exp, &exports, next, next) { + aio_context = exp->ctx; + aio_context_acquire(aio_context); nbd_export_close(exp); + aio_context_release(aio_context); } } From 506902c6fa80210b002e30ff33794bfc718b15c6 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Fri, 20 Sep 2019 17:07:29 -0500 Subject: [PATCH 4/5] tests: Use iothreads during iotest 223 Doing so catches the bugs we just fixed with NBD not properly using correct contexts. Signed-off-by: Eric Blake Message-Id: <20190920220729.31801-1-eblake@redhat.com> --- tests/qemu-iotests/223 | 6 ++++-- tests/qemu-iotests/223.out | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223 index cc48e78ea7..2ba3d8124b 100755 --- a/tests/qemu-iotests/223 +++ b/tests/qemu-iotests/223 @@ -2,7 +2,7 @@ # # Test reading dirty bitmap over NBD # -# Copyright (C) 2018 Red Hat, Inc. +# Copyright (C) 2018-2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -109,7 +109,7 @@ echo echo "=== End dirty bitmaps, and start serving image over NBD ===" echo -_launch_qemu 2> >(_filter_nbd) +_launch_qemu -object iothread,id=io0 2> >(_filter_nbd) # Intentionally provoke some errors as well, to check error handling silent= @@ -117,6 +117,8 @@ _send_qemu_cmd $QEMU_HANDLE '{"execute":"qmp_capabilities"}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"blockdev-add", "arguments":{"driver":"qcow2", "node-name":"n", "file":{"driver":"file", "filename":"'"$TEST_IMG"'"}}}' "return" +_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-blockdev-set-iothread", + "arguments":{"node-name":"n", "iothread":"io0"}}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable", "arguments":{"node":"n", "name":"b"}}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"nbd-server-add", diff --git a/tests/qemu-iotests/223.out b/tests/qemu-iotests/223.out index 5d00398c11..23b34fcd20 100644 --- a/tests/qemu-iotests/223.out +++ b/tests/qemu-iotests/223.out @@ -27,6 +27,7 @@ wrote 2097152/2097152 bytes at offset 2097152 {"return": {}} {"return": {}} {"return": {}} +{"return": {}} {"error": {"class": "GenericError", "desc": "NBD server not running"}} {"return": {}} {"error": {"class": "GenericError", "desc": "NBD server already running"}} From da5e1169183ca6eb6fb470dc32ed1bfc24d1d406 Mon Sep 17 00:00:00 2001 From: Vladimir Sementsov-Ogievskiy Date: Tue, 10 Sep 2019 10:59:43 +0300 Subject: [PATCH 5/5] util/qemu-sockets: fix keep_alive handling in inet_connect_saddr MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In "if (saddr->keep_alive) {" we may already be on error path, with invalid sock < 0. Fix it by returning error earlier. Reported-by: Coverity (CID 1405300) Fixes: aec21d31756cbd Suggested-by: Peter Maydell Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20190910075943.12977-1-vsementsov@virtuozzo.com> Reviewed-by: Daniel P. Berrangé Signed-off-by: Eric Blake --- util/qemu-sockets.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c index 98ff3a1cce..bcc06d0e01 100644 --- a/util/qemu-sockets.c +++ b/util/qemu-sockets.c @@ -461,12 +461,13 @@ int inet_connect_saddr(InetSocketAddress *saddr, Error **errp) } } + freeaddrinfo(res); + if (sock < 0) { error_propagate(errp, local_err); + return sock; } - freeaddrinfo(res); - if (saddr->keep_alive) { int val = 1; int ret = qemu_setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE,