Commit Graph

207 Commits

Author SHA1 Message Date
John Snow 25de7f5012 python/aqmp: fix ConnectError string method
When ConnectError is used to wrap an Exception that was initialized
without an error message, we are treated to a traceback with a rubbish
line like this:

... ConnectError: Failed to establish session:

Correct this to use the name of an exception as a fallback message:

... ConnectError: Failed to establish session: EOFError

Better!

Signed-off-by: John Snow <jsnow@redhat.com>
Reported-by: Thomas Huth <thuth@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Message-id: 20211111143719.2162525-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-16 14:26:36 -05:00
John Snow f26bd6ff21 python/aqmp: Fix disconnect during capabilities negotiation
If we receive ConnectionResetError (ECONNRESET) while attempting to
perform capabilities negotiation -- prior to the establishment of the
async reader/writer tasks -- the disconnect function is not aware that
we are in an error pathway.

As a result, when attempting to close the StreamWriter, we'll see the
same ConnectionResetError that caused us to initiate a disconnect in the
first place, which will cause the disconnect task itself to fail, which
emits a CRITICAL logging event.

I still don't know if there's a smarter way to check to see if an
exception received at this point is "the same" exception as the one that
caused the initial disconnect, but for now the problem can be avoided by
improving the error pathway detection in the exit path.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Message-id: 20211111143719.2162525-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-16 14:26:36 -05:00
Willian Rampazzo bbbd9b6ec6 tests/acceptance: rename tests acceptance to tests avocado
In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211105155354.154864-3-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-11-08 17:00:22 +01:00
John Snow 76cd358671 python, iotests: replace qmp with aqmp
Swap out the synchronous QEMUMonitorProtocol from qemu.qmp with the sync
wrapper from qemu.aqmp instead.

Add an escape hatch in the form of the environment variable
QEMU_PYTHON_LEGACY_QMP which allows you to cajole QEMUMachine into using
the old implementation, proving that both implementations work
concurrently.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-id: 20211026175612.4127598-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-01 11:54:59 -04:00
John Snow f122be6093 python/aqmp: Create sync QMP wrapper for iotests
This is a wrapper around the async QMPClient that mimics the old,
synchronous QEMUMonitorProtocol class. It is designed to be
interchangeable with the old implementation.

It does not, however, attempt to mimic Exception compatibility.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-id: 20211026175612.4127598-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-01 11:54:59 -04:00
John Snow 0f71c9a936 python/aqmp: Remove scary message
The scary message interferes with the iotests output. Coincidentally, if
iotests works by removing this, then it's good evidence that we don't
really need to scare people away from using it.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-id: 20211026175612.4127598-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-01 11:54:59 -04:00
John Snow 49a608b8c2 python/machine: Handle QMP errors on close more meticulously
To use the AQMP backend, Machine just needs to be a little more diligent
about what happens when closing a QMP connection. The operation is no
longer a freebie in the async world; it may return errors encountered in
the async bottom half on incoming message receipt, etc.

(AQMP's disconnect, ultimately, serves as the quiescence point where all
async contexts are gathered together, and any final errors reported at
that point.)

Because async QMP continues to check for messages asynchronously, it's
almost certainly likely that the loop will have exited due to EOF after
issuing the last 'quit' command. That error will ultimately be bubbled
up when attempting to close the QMP connection. The manager class here
then is free to discard it -- if it was expected.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20211026175612.4127598-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-01 11:54:59 -04:00
John Snow b9420e4f4b python/machine: remove has_quit argument
If we spy on the QMP commands instead, we don't need callers to remember
to pass it. Seems like a fair trade-off.

The one slightly weird bit is overloading this instance variable for
wait(), where we use it to mean "don't issue the qmp 'quit'
command". This means that wait() will "fail" if the QEMU process does
not terminate of its own accord.

In most cases, we probably did already actually issue quit -- some
iotests do this -- but in some others, we may be waiting for QEMU to
terminate for some other reason, such as a test wherein we tell the
guest (directly) to shut down.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20211026175612.4127598-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-11-01 11:54:59 -04:00
John Snow c163c723ef python, iotests: remove socket_scm_helper
It's not used anymore, now.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:11 -04:00
John Snow 514d00df5f python/qmp: add send_fd_scm directly to QEMUMonitorProtocol
It turns out you can do this directly from Python ... and because of
this, you don't need to worry about setting the inheritability of the
fds or spawning another process.

Doing this is helpful because it allows QEMUMonitorProtocol to keep its
file descriptor and socket object as private implementation
details. /that/ is helpful in turn because it allows me to write a
compatible, alternative implementation.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:11 -04:00
John Snow d911accf0a python/qmp: clear events on get_events() call
All callers in the tree *already* clear the events after a call to
get_events(). Do it automatically instead and update callsites to remove
the manual clear call.

These semantics are quite a bit easier to emulate with async QMP, and
nobody appears to be abusing some emergent properties of what happens if
you decide not to clear them, so let's dial down to the dumber, simpler
thing.

Specifically: callers of clear() right after a call to get_events() are
more likely expressing their desire to not see any events they just
retrieved, whereas callers of clear_events() not in relation to a recent
call to pull_event/get_events are likely expressing their desire to
simply drop *all* pending events straight onto the floor. In the sync
world, this is safe enough; in the async world it's nearly impossible to
promise that nothing happens between getting and clearing the
events.

Making the retrieval also clear the queue is vastly simpler.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:11 -04:00
John Snow 3a3d84f5ec python/aqmp: Disable logging messages by default
AQMP is a library, and ideally it should not print error diagnostics
unless a user opts into seeing them. By default, Python will print all
WARNING, ERROR or CRITICAL messages to screen if no logging
configuration has been created by a client application.

In AQMP's case, ERROR logging statements are used to report additional
detail about runtime failures that will also eventually be reported to the
client library via an Exception, so these messages should not be
rendered by default.

(Why bother to have them at all, then? In async contexts, there may be
multiple Exceptions and we are only able to report one of them back to
the client application. It is not reasonably easy to predict ahead of
time if one or more of these Exceptions will be squelched. Therefore,
it's useful to log intermediate failures to help make sense of the
ultimate, resulting failure.)

Add a NullHandler that will suppress these messages until a client
application opts into logging via logging.basicConfig or similar. Note
that upon calling basicConfig(), this handler will *not* suppress these
messages from being displayed by the client's configuration.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210923004938.3999963-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:11 -04:00
John Snow 3e55dc35b8 python/aqmp: Reduce severity of EOFError-caused loop terminations
When we encounter an EOFError, we don't know if it's an "error" in the
perspective of the user of the library yet. Therefore, we should not log
it as an error. Reduce the severity of this logging message to "INFO" to
indicate that it's something that we expect to occur during the normal
operation of the library.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210923004938.3999963-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:11 -04:00
John Snow 58026b11f3 python/aqmp: Add dict conversion method to Greeting object
The iotests interface expects to return the greeting as a dict; AQMP
offers it as a rich object.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210923004938.3999963-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:10 -04:00
John Snow 6e2f6ec561 python/aqmp: add send_fd_scm
Add an implementation for send_fd_scm to the async QMP implementation.
Like socket_scm_helper mentions, a non-empty payload is required for
QEMU to process the ancillary data. A space is most useful because it
does not disturb the parsing of subsequent JSON objects.

A note on "voiding the warranty":

Python 3.11 removes support for calling sendmsg directly from a
transport's socket. There is no other interface for doing this, our use
case is, I suspect, "quite unique".

As far as I can tell, this is safe to do -- send_fd_scm is a synchronous
function and we can be guaranteed that the async coroutines will *not* be
running when it is invoked. In testing, it works correctly.

I investigated quite thoroughly the possibility of creating my own
asyncio Transport (The class that ultimately manages the raw socket
object) so that I could manage the socket myself, but this is so wildly
invasive and unportable I scrapped the idea. It would involve a lot of
copy-pasting of various python utilities and classes just to re-create
the same infrastructure, and for extremely little benefit. Nah.

Just boldly void the warranty instead, while I try to follow up on
https://bugs.python.org/issue43232

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210923004938.3999963-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:10 -04:00
John Snow 6bfebc7306 python/aqmp: Return cleared events from EventListener.clear()
This serves two purposes:

(1) It is now possible to discern whether or not clear() removed any
event(s) from the queue with absolute certainty, and

(2) It is now very easy to get a List of all pending events in one
chunk, which is useful for the sync bridge.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:10 -04:00
John Snow 16cce725ed python/aqmp: add .empty() method to EventListener
Synchronous clients may want to know if they're about to block waiting
for an event or not. A method such as this is necessary to implement a
compatible interface for the old QEMUMonitorProtocol using the new async
internals.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:10 -04:00
John Snow 0257209a09 python/aqmp: add greeting property to QMPClient
Expose the greeting as a read-only property of QMPClient so it can be
retrieved at-will.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20210923004938.3999963-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-10-12 12:22:10 -04:00
G S Niteesh Babu 99e45a6131 python/aqmp-tui: Add syntax highlighting
Add syntax highlighting for the incoming and outgoing QMP messages.
This is achieved using the pygments module which was added in a
previous commit.

The current implementation is a really simple one which doesn't
allow for any configuration. In future this has to be improved
to allow for easier theme config using an external config of
some sort.

Signed-off-by: G S Niteesh Babu <niteesh.gs@gmail.com>
Message-Id: <20210823220746.28295-6-niteesh.gs@gmail.com>
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
G S Niteesh Babu aeb6b48a47 python/aqmp-tui: Add AQMP TUI
Added AQMP TUI.

Implements the follwing basic features:
1) Command transmission/reception.
2) Shows events asynchronously.
3) Shows server status in the bottom status bar.
4) Automatic retries on disconnects and error conditions.

Also added type annotations and necessary pylint/mypy configurations.

Signed-off-by: G S Niteesh Babu <niteesh.gs@gmail.com>
Message-Id: <20210823220746.28295-3-niteesh.gs@gmail.com>
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow ed6d4d7a95 python/aqmp: add scary message
Add a warning whenever AQMP is used to steer people gently away from
using it for the time-being.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-24-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow debbabd77f python/aqmp: add asyncio_run compatibility wrapper
As a convenience. It isn't used by the library itself, but it is used by
the test suite. It will also come in handy for users of the library
still on Python 3.6.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-23-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 41f4f92260 python/aqmp: add _raw() execution interface
This is added in anticipation of wanting it for a synchronous wrapper
for the iotest interface. Normally, execute() and execute_msg() both
raise QMP errors in the form of Python exceptions.

Many iotests expect the entire reply as-is. To reduce churn there, add a
private execution interface that will ease transition churn. However, I
do not wish to encourage its use, so it will remain a private interface.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-22-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow e0fea0b3ac python/aqmp: add execute() interfaces
Add execute() and execute_msg().

_execute() is split into _issue() and _reply() halves so that
hypothetical subclasses of QMP that want to support different execution
paradigms can do so.

I anticipate a synchronous interface may have need of separating the
send/reply phases. However, I do not wish to expose that interface here
and want to actively discourage it, so they remain private interfaces.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-21-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 577737be55 python/aqmp: Add message routing to QMP protocol
Add the ability to handle and route messages in qmp_protocol.py. The
interface for actually sending anything still isn't added until next
commit.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow c67d696b85 python/aqmp: add QMP protocol support
The star of our show!

Add most of the QMP protocol, sans support for actually executing
commands. No problem, that happens in the next several commits.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow b3cda213a7 python/aqmp: add QMP event support
This class was designed as a "mix-in" primarily so that the feature
could be given its own treatment in its own python module.

It gets quite a bit too long otherwise.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow ad07299941 python/aqmp: add well-known QMP object models
The QMP spec doesn't define very many objects that are iron-clad in
their format, but there are a few. This module makes it trivial to
validate them without relying on an external third-party library.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 08f98a2231 python/aqmp: add QMP Message format
The Message class is here primarily to serve as a solid type to use for
mypy static typing for unambiguous annotation and documentation.

We can also stuff JSON serialization and deserialization into this class
itself so it can be re-used even outside this infrastructure.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 762bd4d7a7 python/aqmp: add AsyncProtocol._readline() method
This is added as a courtesy: many protocols are line-based, including
QMP. Putting it in AsyncProtocol lets us keep the QMP class
implementation just a pinch more abstract.

(And, if we decide to add a QTEST implementation later, it will need
this, too. (Yes, I have a QTEST implementation.))

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 12c7a57f5b python/aqmp: add _cb_inbound and _cb_outbound logging hooks
Add hooks designed to log/filter incoming/outgoing messages. The primary
intent for these is to be able to support iotests which may want to log
messages with specific filters for reproducible output.

Another use is for plugging into Urwid frameworks; all messages in/out
can be automatically added to a rendering list for the purposes of a
qmp-shell like tool.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 2686ac1316 python/aqmp: add configurable read buffer limit
QMP can transmit some pretty big messages, and the default limit of 64KB
isn't sufficient. Make sure that we can configure it.

Reported-by: G S Niteesh Babu <niteesh.gs@gmail.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 774c64a58d python/aqmp: add AsyncProtocol.accept() method
It's a little messier than connect, because it wasn't designed to accept
*precisely one* connection. Such is life.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 50e533061f python/aqmp: add logging to AsyncProtocol
Give the connection and the reader/writer tasks nicknames, and add
logging statements throughout.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow c1408345af python/aqmp: Add logging utility helpers
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow c58b42e095 python/aqmp: add runstate state machine to AsyncProtocol
This serves a few purposes:

1. Protect interfaces when it's not safe to call them (via @require)

2. Add an interface by which an async client can determine if the state
has changed, for the purposes of connection management.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow 4ccaab0377 python/aqmp: add generic async message-based protocol support
This is the bare minimum that you need to establish a full-duplex async
message-based protocol with Python's asyncio.

The features to be added in forthcoming commits are:

- Runstate tracking
- Logging
- Support for incoming connections via accept()
- _cb_outbound, _cb_inbound message hooks
- _readline() method

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow a07616d612 python/aqmp: add asyncio compatibility wrappers
Python 3.6 does not have all of the goodies that Python 3.7 does, and we
need to support both. Add some compatibility wrappers needed for this
purpose.

(Note: Python 3.6 is EOL December 2021.)

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210915162955.333025-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow fbfb6a37a3 python/aqmp: add error classes
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:29 -04:00
John Snow a093a65567 python/aqmp: add asynchronous QMP (AQMP) subpackage
For now, it's empty! Soon, it won't be.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20210915162955.333025-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27 12:10:27 -04:00
John Snow 5690b4370b python: Update for pylint 2.10
A few new annoyances. Of note is the new warning for an unspecified
encoding when opening a text file, which actually does indicate a
potentially real problem; see
https://www.python.org/dev/peps/pep-0597/#motivation

Use LC_CTYPE to determine an encoding to use for interpreting QEMU's
terminal output. Note that Python states: "language code and encoding
may be None if their values cannot be determined" -- use a platform
default as a backup.

Notes: Passing encoding=None will generate a suppressed warning on
Python 3.10+ that 'None' should not be passed as the encoding
argument. This behavior may be deprecated in the future and the default
switched to be a ubiquitous UTF-8. Opting in to the locale default will
be done by passing the encoding 'locale', but that isn't available in
3.6 through 3.9. Presumably this warning will be unsuppressed some time
prior to the actual switch and we can re-investigate these issues at
that time if necessary.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-id: 20210916182248.721529-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-16 15:03:56 -04:00
Vladimir Sementsov-Ogievskiy 15c3b863ee python:QEMUMachine: template typing for self returning methods
mypy thinks that return value of these methods in subclusses is
QEMUMachine, which is wrong. So, make typing smarter.

Suggested-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210824083856.17408-26-vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 14:03:47 +02:00
Vladimir Sementsov-Ogievskiy 3f3c9b4c9d python/qemu/machine: QEMUMachine: improve qmp() method
We often call qmp() with unpacking dict, like qmp('foo', **{...}).
mypy don't really like it, it thinks that passed unpacked dict is a
positional argument and complains that it type should be bool (because
second argument of qmp() is conv_keys: bool).

Allow passing dict directly, simplifying interface, and giving a way to
satisfy mypy.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20210824083856.17408-25-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 14:03:47 +02:00
Vladimir Sementsov-Ogievskiy c7daa57eb5 python/qemu/machine.py: refactor _qemu_args()
- use shorter construction
 - don't create new dict if not needed
 - drop extra unpacking key-val arguments
 - drop extra default values

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20210824083856.17408-24-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 14:03:47 +02:00
Emanuele Giuseppe Esposito eb7a91d07a qemu-iotests: add option to show qemu binary logs on stdout
Using the flag -p, allow the qemu binary to print to stdout.

Also create the common function _close_qemu_log_file() to
avoid accessing machine.py private fields directly and have
duplicate code.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210809090114.64834-16-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 12:57:31 +02:00
Emanuele Giuseppe Esposito 804f7695e5 python: qemu: pass the wrapper field from QEMUQtestmachine to QEMUMachine
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Message-Id: <20210809090114.64834-4-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 12:57:31 +02:00
Emanuele Giuseppe Esposito e2f948a8b5 python: qemu: add timer parameter for qmp.accept socket
Also add a new _qmp_timer field to the QEMUMachine class.

Let's change the default socket timeout to None, so that if
a subclass needs to add a timer, it can be done by modifying
this private field.

At the same time, restore the timer to be 15 seconds in iotests.py, to
give an upper bound to the QMP monitor test command execution.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210809090114.64834-2-eesposit@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 12:57:31 +02:00
Wainer dos Santos Moschetta 555fe0c2a8 python/qemu: Add args property to the QEMUMachine class
This added the args property to QEMUMachine so that users of the class
can access and handle the list of arguments to be given to the QEMU
binary.

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210430133414.39905-6-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-13 13:24:38 -04:00
Cleber Rosa b306e26ce0 Acceptance Tests: distinguish between temp and logs dir
Logs can be very important to debug issues, and currently QEMUMachine
instances will remove logs that are created under the temporary
directories.

With this change, the stdout and stderr generated by the QEMU process
started by QEMUMachine will always be kept along the test results
directory.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-6-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-13 13:18:50 -04:00
John Snow 5c02c86586 python: Fix broken ReST docstrings
This patch *doesn't* update all of the docstring standards across the
QEMU package directory to make our docstring usage consistent. It
*doesn't* fix the formatting to make it look pretty or reasonable in
generated output. It *does* fix a few small instances where Sphinx would
emit a build warning because of malformed ReST -- If we built our Python
docs with Sphinx.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-30 21:57:08 -04:00
John Snow 82e6517d9d python: Remove global pylint suppressions
These suppressions only apply to a small handful of places. Instead of
disabling them globally, disable them just in the cases where we
need. The design of the machine class grew quite organically with tons
of constructor and class instance variables -- there's little chance of
meaningfully refactoring it in the near term, so just suppress the
warnings for that class.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-30 21:54:04 -04:00
John Snow 7f17908263 python: expose typing information via PEP 561
https://www.python.org/dev/peps/pep-0561/#specification

Create 'py.typed' files in each subpackage that indicate to mypy that
this is a typed module, so that users of any of these packages can use
mypy to check their code as well.

Note: Theoretically it's possible to ditch MANIFEST.in in favor of using
package_data in setup.cfg, but I genuinely could not figure out how to
get it to include things from the *source root* into the *package root*;
only how to include things from each subpackage. I tried!

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-30 21:54:04 -04:00
John Snow 00376d1345 python/qom: Do not use 'err' name at module scope
Pylint updated to 2.9.0 upstream, adding new warnings for things that
re-use the 'err' variable. Luckily, this only breaks the
python-check-tox job, which is allowed to fail as a warning.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-id: 20210629214323.1329806-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-30 21:54:04 -04:00
John Snow 6be7206efc scripts/qmp-shell: move to python/qemu/qmp/qmp_shell.py
The script will be unavailable for a commit or two, which will help
preserve development history attached to the new file. A forwarder will
be added shortly afterwards.

With qmp_shell in the python qemu.qmp package, now it is fully type
checked, linted, etc. via the Python CI. It will be quite a bit harder
to accidentally break it again in the future.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-41-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:07 -04:00
John Snow eac8aabc92 python/qmp: return generic type from context manager
__enter__ can be invoked from a subclass, so it needs a more flexible
type.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-31-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:07 -04:00
John Snow 1acde76328 python/qmp: add QMPObject type alias
This is meant to represent any generic object seen in a QMPMessage, not
just the root object itself.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-27-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:07 -04:00
John Snow 7e7c2a0de7 python/qemu-ga-client: add entry point
Remove the shebang, and add a package-defined entry point instead. Now,
it can be accessed using 'qemu-ga-client' from the command line after
installing the package.

The next commit adds a forwarder shim that allows the running of this
script without needing to install the package again.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 1e129afc31 scripts/qemu-ga-client: move to python/qemu/qmp/qemu_ga_client.py
The script itself will be unavailable for a few commits before being
restored, with no way to run it right after this commit. This helps move
git history into the new file. To prevent linter regressions, though, we
do need to immediately touch up the filename to remove dashes (to make
the module importable), and remove the executable bit.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 1f6399393b python/qmp: Correct type of QMPReturnValue
It's only a Dict[str, Any] most of the time. It's not actually
guaranteed to be anything in particular. Fix this type to be
more accurate to the reality we live in.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 176c549072 python/qmp: add fuse command to 'qom' tools
The 'fuse' command will be unavailable if 'fusepy' is not installed. It
will simply not load and subsequently be unavailable as a subcommand.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 173d185de9 scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py
Move qom-fuse over to the python package now that it passes the
linter. Update the import paradigms so that it continues to pass in the
context of the Python package.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow c750c02891 python/qmp: Add qom script rewrites
Inspired by qom-set, qom-get, qom-tree and qom-list; combine all four of
those scripts into a single script.

A later addition of qom-fuse as an 'extension' necessitates that some
common features are split out and shared between them.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 587adaca55 python/qmp: add parse_address classmethod
This takes the place of qmp-shell's __get_address function. It also
allows other utilities to share the same parser and syntax for
specifying QMP locations.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 5d15c9b875 python/qmp: Fix type of SocketAddrT
In porting the qom tools, qmp-shell, etc; it becomes evident that this
type is wrong.

This is an integer, not a string. We didn't catch this before because
none of QEMUMonitorProtocol's *users* happen to be checked, and the
internal logic of this class is otherwise self-consistent. Additionally,
mypy was not introspecting into the socket() interface to realize we
were passing a bad type for AF_INET. Fixed now.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-18 16:10:06 -04:00
John Snow 81f8c4467c python: move flake8 config to setup.cfg
Update the comment concerning the flake8 exception to match commit
42c0dd12, whose commit message stated:

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-19-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow ef42440d79 python: move pylintrc into setup.cfg
Delete the empty settings now that it's sharing a home with settings for
other tools.

pylint can now be run from this folder as "pylint qemu".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow d1e0476958 python: add pylint import exceptions
Pylint 2.5.x - 2.7.x have regressions that make import checking
inconsistent, see:

https://github.com/PyCQA/pylint/issues/3609
https://github.com/PyCQA/pylint/issues/3624
https://github.com/PyCQA/pylint/issues/3651

Pinning to 2.4.4 is worse, because it mandates versions of shared
dependencies that are too old for features we want in isort and mypy.
Oh well.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow 93128815af python: add directory structure README.rst files
Add short readmes to python/, python/qemu/, python/qemu/machine,
python/qemu/qmp, and python/qemu/utils that explain the directory
hierarchy. These readmes are visible when browsing the source on
e.g. gitlab/github and are designed to help new developers/users quickly
make sense of the source tree.

They are not designed for inclusion in a published manual.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow beb6b57b3b python: create qemu packages
move python/qemu/*.py to python/qemu/[machine, qmp, utils]/*.py and
update import directives across the tree.

This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go into new 'machine',
'qmp' and 'utils' subpackages instead.

Implement machine/__init__.py making the top-level classes and functions
from its various modules available directly inside the package. Change
qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
library classes are available directly from "qemu.qmp" instead of
"qemu.qmp.qmp".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow 859aeb67d7 python/machine: Trim line length to below 80 chars
One more little delinting fix that snuck in.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow a0eae17a59 python/machine: disable warning for Popen in _launch()
We handle this resource rather meticulously in
shutdown/kill/wait/__exit__ et al, through the laborious mechanisms in
_do_shutdown().

Quiet this pylint warning here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-7-jsnow@redhat.com
Message-id: 20210517184808.3562549-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow 63c33f3c28 python/machine: Disable pylint warning for open() in _pre_launch
Shift the open() call later so that the pylint pragma applies *only* to
that one open() call. Add a note that suggests why this is safe: the
resource is unconditionally cleaned up in _post_shutdown().

_post_shutdown is called after failed launches (see launch()), and
unconditionally after every call to shutdown(), and therefore also on
__exit__.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-6-jsnow@redhat.com
Message-id: 20210517184808.3562549-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow 8825fed82a python/console_socket: Add a pylint ignore
We manage cleaning up this resource ourselves. Pylint should shush.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-5-jsnow@redhat.com
Message-id: 20210517184808.3562549-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow 14b41797d5 python/machine: use subprocess.run instead of subprocess.Popen
use run() instead of Popen() -- to assert to pylint that we are not
forgetting to close a long-running program.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-4-jsnow@redhat.com
Message-id: 20210517184808.3562549-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow 07b71233a7 python/machine: use subprocess.DEVNULL instead of open(os.path.devnull)
One less file resource to manage, and it helps quiet some pylint >=
2.8.0 warnings about not using a with-context manager for the open call.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-3-jsnow@redhat.com
Message-id: 20210517184808.3562549-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
John Snow ee1a27235b python/console_socket: avoid one-letter variable
Fixes pylint warnings.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210527211715.394144-2-jsnow@redhat.com
Message-id: 20210517184808.3562549-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
Cleber Rosa 976218cbe7 Python: add utility function for retrieving port redirection
Slightly different versions for the same utility code are currently
present on different locations.  This unifies them all, giving
preference to the version from virtiofs_submounts.py, because of the
last tweaks added to it.

While at it, this adds a "qemu.utils" module to host the utility
function and a test.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-4-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
[Squashed in below fix. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210601154546.130870-2-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:20 -04:00
Cleber Rosa 2ca6e26cea Python: expose QEMUMachine's temporary directory
Each instance of qemu.machine.QEMUMachine currently has a "test
directory", which may not have any relation to a "test", and it's
really a temporary directory.

Users instantiating the QEMUMachine class will be able to set the
location of the directory that will *contain* the QEMUMachine unique
temporary directory, so that parameter name has been changed from
test_dir to base_temp_dir.

A property has been added to allow users to access it without using
private attributes, and with that, the directory is created on first
use of the property.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-3-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:20 -04:00
Cleber Rosa 3c1e16c60c Python: close the log file kept by QEMUMachine before reading it
Closing a file that is open for writing, and then reading from it
sounds like a better idea than the opposite, given that the content
will be flushed.

Reference: https://docs.python.org/3/library/io.html#io.IOBase.close
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-2-crosa@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-02-15 21:40:16 -05:00
Alex Bennée afded359a6 python: add __repr__ to ConsoleSocket to aid debugging
While attempting to debug some console weirdness I thought it would be
worth making it easier to see what it had inside.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20201210190417.31673-6-alex.bennee@linaro.org>
2021-01-02 21:03:09 +01:00
Paolo Bonzini 991c180d74 treewide: do not use short-form boolean options
They are going to be deprecated, avoid warnings on stdout while the
tests run.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:11 -05:00
Alex Bennée 8c175c63ee tests: add prefixes to the bare mkdtemp calls
The first step to debug a thing is to know what created the thing in
the first place. Add some prefixes so random tmpdir's have something
grep in the code.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201117173635.29101-3-alex.bennee@linaro.org>
2020-11-23 09:51:43 +00:00
John Snow 39cf73c349 python/qemu/qmp.py: Fix settimeout operation
We enabled callers to interface directly with settimeout, but this
reacts poorly with blocking/nonblocking operation; as they are using the
same internal mechanism.

1. Whenever we change the blocking mechanism temporarily, always set it
back to what it was afterwards.

2. Disallow callers from setting a timeout of "0", which means
Non-blocking mode. This is going to create more weird problems than
anybody wants, so just forbid it.

I opt not to coerce '0' to 'None' to maintain the principal of least
surprise in mirroring the semantics of Python's interface.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201009175123.249009-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 15:00:06 -04:00
John Snow d5cca076c3 python/qemu/qmp.py: re-raise OSError when encountered
Nested if conditions don't change when the exception block fires; we
need to explicitly re-raise the error if we didn't intend to capture and
suppress it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201009175123.249009-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 14:58:47 -04:00
John Snow d2b08b79b0 python/qemu/qmp.py: Preserve error context on re-raise
Use the "from ..." phrasing when re-raising errors to preserve their
initial context, to help aid debugging when things go wrong.

This also silences a pylint 2.6.0+ error.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow af0db88254 python/qemu/console_socket.py: avoid encoding to/from string
We can work directly in bytes instead of translating back and forth to
string, which removes the question of which encodings to use.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow e35c138267 python/qemu/console_socket.py: Add type hint annotations
Finish the typing of console_socket.py with annotations and no code
changes.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow 714ac05a19 python/qemu/console_socket.py: Clarify type of drain_thread
Mypy needs just a little help to guess the type here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow 6cf4cce7cb python/qemu/console_socket.py: fix typing of settimeout
The types and names of the parameters must match the socket.socket interface.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow ff3513e632 python/qemu/console_socket.py: Correct type of recv()
The type and parameter names of recv() should match socket.socket().

OK, easy enough, but in the cases we don't pass straight through to the
real socket implementation, we probably can't accept such flags. OK, for
now, assert that we don't receive flags in such cases.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow f12a282ff4 python/qemu: Add mypy type annotations
These should all be purely annotations with no changes in behavior at
all. You need to be in the python folder, but you should be able to
confirm that these annotations are correct (or at least self-consistent)
by running `mypy --strict qemu`.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow aad3f3bb6c python/qemu: make 'args' style arguments immutable
These arguments don't need to be mutable and aren't really used as
such. Clarify their types as immutable and adjust code to match where
necessary.

In general, It's probably best not to accept a user-defined mutable
object and store it as internal object state unless there's a strong
justification for doing so. Instead, try to use generic types as input
with empty tuples as the default, and coerce to list where necessary.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201006235817.3280413-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow 9223fda464 python/machine.py: fix _popen access
As always, Optional[T] causes problems with unchecked access. Add a
helper that asserts the pipe is present before we attempt to talk with
it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow be1183e52f python/machine.py: Add _qmp access shim
Like many other Optional[] types, it's not always a given that this
object will be set. Wrap it in a type-shim that raises a meaningful
error and will always return a concrete type.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow aaa81ec609 python/machine.py: use qmp.command
machine.py and qmp.py both do the same thing here; refactor machine.py
to use qmp.py's functionality more directly.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20201006235817.3280413-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow 1847a4a8c2 python/machine.py: Handle None events in events_wait
If the timeout is 0, we can get None back. Handle this explicitly.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow 652809dfa6 python/machine.py: Don't modify state in _base_args()
Don't append to the _remove_files list during _base_args; instead do so
during _launch. Rework _base_args as a @property to help facilitate
this impression.

This has the additional benefit of making the type of _console_address
easier to analyze statically.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow c5e61a6da8 python/machine.py: reorder __init__
Put the init arg handling all at the top, and mostly in order (deviating
when one is dependent on another), and put what is effectively runtime
state declaration at the bottom.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow c4e6023f05 python/machine.py: Fix monitor address typing
Prior to this, it's difficult for mypy to intuit what the concrete type
of the monitor address is; it has difficulty inferring the type across
two variables.

Create _monitor_address as a property that always returns a valid
address to simplify static type analysis.

To preserve our ability to clean up, use a simple boolean to indicate
whether or not we should try to clean up the sock file after execution.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00
John Snow 932ca4bbde python/qemu: use isort to lay out imports
Borrowed from the QAPI cleanup series, use the same configuration to
standardize the way we write and sort imports.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2020-10-20 09:37:57 -04:00