Commit Graph

61 Commits

Author SHA1 Message Date
Peter Maydell
4fd1cbaf14 Monitor patches for 2018-07-03
-----BEGIN PGP SIGNATURE-----
 
 iQIcBAABAgAGBQJbO+iNAAoJEDhwtADrkYZTX9gQAJFT0DPtUYQsqbYAhrUIJo/a
 lT5OAnMnMou6Qu0XHjSYrqdZu7Ega00rvyeYgtB54lmLcOVrjAxF0SwcPcoOuz7K
 O9o00ZmwuFPUugJuYBLDRqJryejbZTCGSOg3cp+YlV6naW6Omck97iu7G/MZWLvB
 FJvIIUaPIAGklRhSodhC+yePp1PNbfGVcpDjAxn4e1UQ/2M6gwSFx9iPbNl2WotN
 CXjRHmkB4ZcxwcDtzzFLwkXTDkryZE27sOgFjDG5xFWO/oCS+Px7a0vME8YM+j2r
 eNa93g16ZohNqHb9t6GarlqrqNqbQF2t6JAPpmlIX5qb3Dtb87MQ/ExdN+FJ4btD
 IfBkTnXbs7qkZpConRlrhdd2J01ChnCdcmddUeenlq+8mIMjQJXH2RCwRVKyoZYs
 4Z74XWy2A+xMJgjqY9WO7E4jy4QVDSckKIm6je/O8fXzXWxthi4H1/2EN5nAv1hY
 zNxP/SW0vpqiR/cVQAr2N6VGz2/m9cWQIf0NwFc765gypPBbCL5HLrb+/xC86Bi3
 ylg9YlTw84oXtpSauTYu3oE989iDndu/s9BEUe5fW5g+8azs+h5SHVTwq0Y8dzSZ
 n2FsGxr76BlOIu5i1oIYNUUhAoU3txddueT4SNHE5Iaz1x6fAKnMP2hxoo5DWDzn
 1Rzgq4ATB8H+V6Og942n
 =/2SW
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2018-07-03-v2' into staging

Monitor patches for 2018-07-03

# gpg: Signature made Tue 03 Jul 2018 22:20:13 BST
# gpg:                using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-monitor-2018-07-03-v2: (32 commits)
  qapi: Polish command flags documentation in qapi-code-gen.txt
  monitor: Improve some comments
  qmp: Clean up capability negotiation after commit 02130314d8
  qobject: Let qobject_from_jsonf() fail instead of abort
  qmp: Switch timestamp_put() to qdict_from_jsonf_nofail()
  qmp: Add some comments around null responses
  qmp: Simplify monitor_qmp_respond()
  qmp: Replace get_qmp_greeting() by qmp_greeting()
  qmp: Replace monitor_json_emitter{,raw}() by qmp_{queue,send}_response()
  qmp: Use QDict * instead of QObject * for response objects
  qmp: De-duplicate error response building
  qobject: New qdict_from_jsonf_nofail()
  monitor: Peel off @mon_global wrapper
  monitor: Rename use_io_thr to use_io_thread
  qmp: Don't let JSON errors jump the queue
  qmp: Don't let malformed in-band commands jump the queue
  tests/qmp-test: Demonstrate QMP errors jumping the queue
  qmp: Simplify code around monitor_qmp_dispatch_one()
  qmp: Always free QMPRequest with qmp_request_free()
  qmp: Revert change to handle_qmp_command tracepoint
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2018-07-05 11:25:14 +01:00
Marc-André Lureau
967c885108 qapi: add 'if' to top-level expressions
Accept 'if' key in top-level elements, accepted as string or list of
string type. The following patches will modify the test visitor to
check the value is correctly saved, and generate #if/#endif code (as a
single #if/endif line or a series for a list).

Example of 'if' key:
{ 'struct': 'TestIfStruct', 'data': { 'foo': 'int' },
  'if': 'defined(TEST_IF_STRUCT)' }

The generated code is for now *unconditional*. Later patches generate
the conditionals.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-2-marcandre.lureau@redhat.com>
[Commit message and Documentation improved]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2018-07-03 18:21:24 +02:00
Markus Armbruster
c069821220 qmp: Say "out-of-band" instead of "Out-Of-Band"
Affects documentation and a few error messages.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180703085358.13941-2-armbru@redhat.com>
2018-07-03 11:46:54 +02:00
Anton Nefedov
800877bb16 qapi: allow empty branches in flat unions
It often happens that just a few discriminator values imply extra data in
a flat union. Existing checks did not make possible to leave other values
uncovered. Such cases had to be worked around by either stating a dummy
(empty) type or introducing another (subset) discriminator enumeration.

Both options create redundant entities in qapi files for little profit.

With this patch it is not necessary anymore to add designated union
fields for every possible value of a discriminator enumeration.

Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
Message-Id: <1529311206-76847-2-git-send-email-anton.nefedov@virtuozzo.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2018-06-22 16:33:46 +02:00
Igor Mammedov
7b13f2c27a tests: qapi-schema tests for allow-preconfig
use new allow-preconfig parameter in tests and make sure that
the QAPISchema can parse allow-preconfig correctly

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <1526058959-41425-1-git-send-email-imammedo@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-05-30 13:19:14 -03:00
Peter Xu
1a1b11dc0f tests: add oob-test for qapi-schema
It simply tests the new OOB capability, and make sure the QAPISchema can
parse it correctly.

Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180326063901.27425-7-peterx@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-27 10:17:45 -05:00
Markus Armbruster
4d2d5c41a9 qapi: Introduce a first class 'null' type
I expect the 'null' type to be useful mostly for members of alternate
types.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2017-07-24 13:35:11 +02:00
Marc-André Lureau
7c877c8030 tests: Remove test cases for alternates of 'number' and 'int'
Alternates with both a 'number' and an 'int' branch will become
invalid when the next patch merges of QFloat and QInt into QNum.
More sophisticated alternate code could keep them valid, but since
we have no users outside tests, simply drop the tests.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20170607163635.17635-4-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2017-06-19 14:56:29 +02:00
Markus Armbruster
c0644771eb qapi: Reject alternates that can't work with keyval_parse()
Alternates are sum types like unions, but use the JSON type on the
wire / QType in QObject instead of an explicit tag.  That's why we
require alternate members to have distinct QTypes.

The recently introduced keyval_parse() (commit d454dbe) can only
produce string scalars.  The qobject_input_visitor_new_keyval() input
visitor mostly hides the difference, so code using a QObject input
visitor doesn't have to care whether its input was parsed from JSON or
KEY=VALUE,...  The difference leaks for alternates, as noted in commit
0ee9ae7: a non-string, non-enum scalar alternate value can't currently
be expressed.

In part, this is just our insufficiently sophisticated implementation.
Consider alternate type 'GuestFileWhence'.  It has an integer member
and a 'QGASeek' member.  The latter is an enumeration with values
'set', 'cur', 'end'.  The meaning of b=set, b=cur, b=end, b=0, b=1 and
so forth is perfectly obvious.  However, our current implementation
falls apart at run time for b=0, b=1, and so forth.  Fixable, but not
today; add a test case and a TODO comment.

Now consider an alternate type with a string and an integer member.
What's the meaning of a=42?  Is it the string "42" or the integer 42?
Whichever meaning you pick makes the other inexpressible.  This isn't
just an implementation problem, it's fundamental.  Our current
implementation will pick string.

So far, we haven't needed such alternates.  To make sure we stop and
think before we add one that cannot sanely work with keyval_parse(),
let's require alternate members to have sufficiently distinct
representation in KEY=VALUE,... syntax:

* A string member clashes with any other scalar member

* An enumeration member clashes with bool members when it has value
  'on' or 'off'.

* An enumeration member clashes with numeric members when it has a
  value that starts with '-', '+', or a decimal digit.  This is a
  rather lazy approximation of the actual number syntax accepted by
  the visitor.

  Note that enumeration values starting with '-' and '+' are rejected
  elsewhere already, but better safe than sorry.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <1495471335-23707-5-git-send-email-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
2017-05-31 16:04:09 +02:00
Markus Armbruster
8168ca8ea3 tests/qapi-schema: Avoid 'str' in alternate test cases
The next commit is going to make alternate members of type 'str'
conflict with other scalar types.  Would break a few test cases that
don't actually require 'str'.  Flip them from 'str' to 'bool' or
'EnumOne'.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <1495471335-23707-4-git-send-email-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
2017-05-31 16:04:05 +02:00
Markus Armbruster
1554a8fae9 qapi: Have each QAPI schema declare its returns white-list
qapi.py has a hardcoded white-list of command names that may violate
the rules on permitted return types.  Add a new pragma directive
'returns-whitelist', and use it to replace the hard-coded white-list.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <1489582656-31133-6-git-send-email-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2017-03-16 07:13:02 +01:00
Markus Armbruster
87c16dceca qapi: Back out doc comments added just to please qapi.py
This reverts commit 3313b61's changes to tests/qapi-schema/, except
for tests/qapi-schema/doc-*.

We could keep some of these doc comments to serve as positive test
cases.  However, they don't actually add to what we get from doc
comment use in actual schemas, as we we don't test output matches
expectations, and don't systematically cover doc comment features.
Proper positive test coverage would be nice.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <1489582656-31133-4-git-send-email-armbru@redhat.com>
2017-03-16 07:13:01 +01:00
Marc-André Lureau
3313b6124b qapi: add qapi2texi script
As the name suggests, the qapi2texi script converts JSON QAPI
description into a texi file suitable for different target
formats (info/man/txt/pdf/html...).

It parses the following kind of blocks:

Free-form:

  ##
  # = Section
  # == Subsection
  #
  # Some text foo with *emphasis*
  # 1. with a list
  # 2. like that
  #
  # And some code:
  # | $ echo foo
  # | -> do this
  # | <- get that
  #
  ##

Symbol description:

  ##
  # @symbol:
  #
  # Symbol body ditto ergo sum. Foo bar
  # baz ding.
  #
  # @param1: the frob to frobnicate
  # @param2: #optional how hard to frobnicate
  #
  # Returns: the frobnicated frob.
  #          If frob isn't frobnicatable, GenericError.
  #
  # Since: version
  # Notes: notes, comments can have
  #        - itemized list
  #        - like this
  #
  # Example:
  #
  # -> { "execute": "quit" }
  # <- { "return": {} }
  #
  ##

That's roughly following the following EBNF grammar:

api_comment = "##\n" comment "##\n"
comment = freeform_comment | symbol_comment
freeform_comment = { "# " text "\n" | "#\n" }
symbol_comment = "# @" name ":\n" { member | tag_section | freeform_comment }
member = "# @" name ':' [ text ] "\n" freeform_comment
tag_section = "# " ( "Returns:", "Since:", "Note:", "Notes:", "Example:", "Examples:" ) [ text ]  "\n" freeform_comment
text = free text with markup

Note that the grammar is ambiguous: a line "# @foo:\n" can be parsed
both as freeform_comment and as symbol_comment.  The actual parser
recognizes symbol_comment.

See docs/qapi-code-gen.txt for more details.

Deficiencies and limitations:
- the generated QMP documentation includes internal types
- union type support is lacking
- type information is lacking in generated documentation
- doc comment error message positions are imprecise, they point
  to the beginning of the comment.
- a few minor issues, all marked TODO/FIXME in the code

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20170113144135.5150-16-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[test-qapi.py tweaked to avoid trailing empty lines in .out]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2017-01-16 10:10:35 +01:00
Eric Blake
c818408e44 qapi: Implement boxed types for commands/events
Turn on the ability to pass command and event arguments in
a single boxed parameter, which must name a non-empty type
(although the type can be a struct with all optional members).
For structs, it makes it possible to pass a single qapi type
instead of a breakout of all struct members (useful if the
arguments are already in a struct or if the number of members
is large); for other complex types, it is now possible to use
a union or alternate as the data for a command or event.

The empty type may be technically feasible if needed down the
road, but it's easier to forbid it now and relax things to allow
it later, than it is to allow it now and have to special case
how the generated 'q_empty' type is handled (see commit 7ce106a9
for reasons why nothing is generated for the empty type).  An
alternate type is never considered empty, but now that a boxed
type can be either an object or an alternate, we have to provide
a trivial QAPISchemaAlternateType.is_empty().  The new call to
arg_type.is_empty() during QAPISchemaCommand.check() requires
that we first check the type in question; but there is no chance
of introducing a cycle since objects do not refer back to commands.

We still have a split in syntax checking between ad-hoc parsing
up front (merely validates that 'boxed' has a sane value) and
during .check() methods (if 'boxed' is set, then 'data' must name
a non-empty user-defined type).

Generated code is unchanged, as long as no client uses the
new feature.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1468468228-27827-10-git-send-email-eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Test files renamed to *-boxed-*]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-07-19 13:21:08 +02:00
Eric Blake
ac4338f8eb qapi: Allow anonymous base for flat union
Rather than requiring all flat unions to explicitly create
a separate base struct, we can allow the qapi schema to specify
the common members via an inline dictionary. This is similar to
how commands can specify an inline anonymous type for its 'data'.
We already have several struct types that only exist to serve as
a single flat union's base; the next commit will clean them up.
In particular, this patch's change to the BlockdevOptions example
in qapi-code-gen.txt will actually be done in the real QAPI schema.

Now that anonymous bases are legal, we need to rework the
flat-union-bad-base negative test (as previously written, it
forms what is now valid QAPI; tweak it to now provide coverage
of a new error message path), and add a positive test in
qapi-schema-test to use an anonymous base (making the integer
argument optional, for even more coverage).

Note that this patch only allows anonymous bases for flat unions;
simple unions are already enough syntactic sugar that we do not
want to burden them further.  Meanwhile, while it would be easy
to also allow an anonymous base for structs, that would be quite
redundant, as the members can be put right into the struct
instead.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1458254921-17042-15-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-03-18 10:29:26 +01:00
Eric Blake
972a110162 qapi: Fix command with named empty argument type
The generator special-cased

 { 'command':'foo', 'data': {} }

to avoid emitting a visitor variable, but failed to see that

 { 'struct':'NamedEmptyType, 'data': {} }
 { 'command':'foo', 'data':'NamedEmptyType' }

needs the same treatment.  There, the generator happily generates a
visitor to get no arguments, and a visitor to destroy no arguments;
and the compiler isn't happy with that, as demonstrated by the updated
qapi-schema-test.json:

  tests/test-qmp-marshal.c: In function ‘qmp_marshal_user_def_cmd0’:
  tests/test-qmp-marshal.c:264:14: error: variable ‘v’ set but not used [-Werror=unused-but-set-variable]
       Visitor *v;
                ^

No change to generated code except for the testsuite addition.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1458254921-17042-3-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-03-18 10:29:25 +01:00
Eric Blake
14f00c6c49 qapi: Rename 'fields' to 'members' in generator
C types and JSON objects don't have fields, but members.  We
shouldn't gratuitously invent terminology.  This patch is a
strict renaming of generator code internals (including testsuite
comments), before later patches rename C interfaces.

No change to generated code with this patch.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1457021813-10704-2-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-03-05 10:40:52 +01:00
Eric Blake
68d078395d qapi: Add tests of complex objects within alternate
Upcoming patches will adjust how we visit an object branch of an
alternate; but we were completely lacking testsuite coverage.
Rectify this, so that the future patches will be able to highlight
the changes and still prove that we avoided regressions.

In particular, the use of a flat union UserDefFlatUnion rather
than a simple struct UserDefA as the branch will give us coverage
of an object with variants.  And visiting an alternate as both
the top level and as a nested member gives confidence in correct
memory allocation handling, especially if the test is run under
valgrind.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1455778109-6278-5-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-02-19 11:08:57 +01:00
Eric Blake
9d3524b39e qapi-visit: Honor prefix of discriminator enum
When we added support for a user-specified prefix for an enum
type (commit 351d36e), we forgot to teach the qapi-visit code
to honor that prefix in the case of using a prefixed enum as
the discriminator for a flat union.  While there is still some
on-list debate on whether we want to keep prefixes, we should
at least make it work as long as it is still part of the code
base.

Reported-by: Daniel P. Berrange <berrange@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1455665965-27638-1-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-02-19 11:08:56 +01:00
Eric Blake
c43567c120 qapi: Fix c_name() munging
The method c_name() is supposed to do two different actions: munge
'-' into '_', and add a 'q_' prefix to ticklish names.  But it did
these steps out of order, making it possible to submit input that
is not ticklish until after munging, where the output then lacked
the desired prefix.

The failure is exposed easily if you have a compiler that recognizes
C11 keywords, and try to name a member '_Thread-local', as it would
result in trying to compile the declaration 'uint64_t _Thread_local;'
which is not valid.  However, this name violates our conventions
(ultimately, want to enforce that no qapi names start with single
underscore), so the test is slightly weaker by instead testing
'wchar-t'; the declaration 'uint64_t wchar_t;' is valid in C (where
wchar_t is only a typedef) but would fail with a C++ compiler (where
it is a keyword).

Fix things by reversing the order of actions within c_name().

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1447836791-369-18-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-12-17 08:21:27 +01:00
Eric Blake
748053c97b qapi: Use generated TestStruct machinery in tests
Commit d88f5fd and friends first introduced the various test-qmp-*
tests in 2011, with duplicated hand-rolled TestStruct machinery,
to make sure the qapi visitor interface was tested.  Later, commit
4f193e3 in 2013 added a .json file for further testing use by the
files, but without consolidating any of the existing hand-rolled
visitors.  And with four copies, subtle differences have crept in,
between the tests themselves (mainly whitespace differences, but
also a question of whether to use NULL or "TestStruct" when
calling visit_start_struct()) and from what the generator produces
(the hand-rolled versions did not cater to partially-allocated
objects, because they did not have a deallocation usage).

Of course, just because the visitor interface is tested does not
mean it is a sane interface; and future patches will be changing
some of the visitor contracts.  Rather than having to duplicate
the cleanup work in each copy of the TestStruct visitor, and keep
each hand-rolled copy in sync with what the generator supplies, we
might as well just test what the generator should give us in the
first place.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1446791754-23823-2-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-11-09 16:45:05 +01:00
Eric Blake
1976708321 tests/qapi-schema: Test for reserved names, empty struct
Add some testsuite coverage to ensure future patches are on
the right track:

Our current C representation of qapi arrays is done by appending
'List' to the element name; but we are not preventing the
creation of an object type with the same name.  Add
reserved-type-list.json to test this.  Then rename
enum-union-clash.json to reserved-type-kind.json to cover the
reservation that we DO detect, and shorten it to match the fact
that the name is reserved even if there is no clash.

We are failing to detect a collision between a dictionary member
and the implicit 'has_*' flag for another optional member. The
easiest fix would be for a future patch to reserve the entire
"has[-_]" namespace for member names (the collision is also
possible for branch names within flat unions, but only as long as
branch names can collide with (non-variant) members; however,
since future patches are about to remove that, it is not worth
testing here). Add reserved-member-has.json to test this.

A similar collision exists between a dictionary member where
c_name() munges what might otherwise be a reserved name to start
with 'q_', and another member explicitly starts with "q[-_]".
Again, the easiest solution for a future patch will be reserving
the entire namespace, but here for commands as well as members.
Add reserved-member-q.json and reserved-command-q.json to test
this; separate tests since arguably our munging of command 'unix'
to 'qmp_q_unix()' could be done without a q_, which is different
than the munging of a member 'unix' to 'foo.q_unix'.

Finally, our testsuite does not have any compilation coverage
of struct inheritance with empty qapi structs.  Update
qapi-schema-test.json to test this.

Note that there is currently no technical reason to forbid type
name patterns from member names, or member name patterns from
types, since the two are not in the same namespace in C and
won't collide; but it's not worth adding positive tests of these
corner cases at this time, especially while there is other churn
pending in patches that rearrange which collisions actually
happen.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1445898903-12082-2-git-send-email-eblake@redhat.com>
[Commit message tweaked slightly]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-11-02 08:30:25 +01:00
Eric Blake
9f08c8ec73 qapi: Lazy creation of array types
Commit ac88219a had several TODO markers about whether we needed
to automatically create the corresponding array type alongside
any other type.  It turns out that most of the time, we don't!

There are a few exceptions: 1) We have a few situations where we
use an array type in internal code but do not expose that type
through QMP; fix it by declaring a dummy type that forces the
generator to see that we want to use the array type.

2) The builtin arrays (such as intList for QAPI ['int']) must
always be generated, because of the way our QAPI_TYPES_BUILTIN
compile guard works: we have situations (at the very least
tests/test-qmp-output-visitor.c) that include both top-level
"qapi-types.h" (via "error.h") and a secondary
"test-qapi-types.h". If we were to only emit the builtin types
when used locally, then the first .h file would not include all
types, but the second .h does not declare anything at all because
the first .h set QAPI_TYPES_BUILTIN, and we would end up with
compilation error due to things like unknown type 'int8List'.

Actually, we may need to revisit how we do type guards, and
change from a single QAPI_TYPES_BUILTIN over to a different
usage pattern that does one #ifdef per qapi type - right now,
the only types that are declared multiple times between two qapi
.json files for inclusion by a single .c file happen to be the
builtin arrays.  But now that we have QAPI 'include' statements,
it is logical to assume that we will soon reach a point where
we want to reuse non-builtin types (yes, I'm thinking about what
it will take to add introspection to QGA, where we will want to
reuse the SchemaInfo type and friends).  One #ifdef per type
will help ensure that generating the same qapi type into more
than one qapi-types.h won't cause collisions when both are
included in the same .c file; but we also have to solve how to
avoid creating duplicate qapi-types.c entry points.  So that
is a problem left for another day.

Generated code for qapi-types and qapi-visit is drastically
reduced; less than a third of the arrays that were blindly
created were actually needed (a quick grep shows we dropped
from 219 to 69 *List types), and the .o files lost more than
30% of their bulk.  [For best results, diff the generated
files with 'git diff --patience --no-index pre post'.]

Interestingly, the introspection output is unchanged - this is
because we already cull all types that are not indirectly
reachable from a command or event, so introspection was already
using only a subset of array types.  The subset of types
introspected is now a much larger percentage of the overall set
of array types emitted in qapi-types.h (since the larger set
shrunk), but still not 100% (evidence that the array types
emitted for our new Dummy structs, and the new struct itself,
don't affect QMP).

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1444710158-8723-9-git-send-email-eblake@redhat.com>
[Moved array info tracking to a later patch]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-10-15 08:39:08 +02:00
Eric Blake
70478cef83 qapi: Drop redundant flat-union-reverse-define test
As of commit 8c3f8e77, we test compilation of forward references
for a struct base type (UserDefOne), flat union base type
(UserDefUnionBase), and flat union branch type
(UserDefFlatUnion2). The only remaining forward reference being
tested for parsing in flat-union-reverse-define was a forward
enum declaration.  Once we make sure that always compiles,
the smaller parse-only test is redundant and can be deleted.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1444710158-8723-7-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-10-15 08:39:07 +02:00
Eric Blake
cae95eae62 qapi: Drop redundant returns-int test
qapi-schema-test was already testing that we could have a
command returning int, but burned a command name in the whitelist.
Merge the redundant positive test returns-int, and pick a name
that reduces the whitelist size.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1444710158-8723-6-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-10-15 08:39:07 +02:00
Eric Blake
625b251c69 qapi: Move empty-enum to compile-time test
Rather than just asserting that we can parse an empty enum,
let's also make sure we can compile it, by including it in
qapi-schema-test.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1444710158-8723-5-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-10-15 08:39:07 +02:00
Eric Blake
9c51b44129 qapi: Test use of 'number' within alternates
Add some testsuite exposure for use of a 'number' as part of
an alternate.  The current state of the tree has a few bugs
exposed by this: our input parser depends on the ordering of
how the qapi schema declared the alternate, and the parser
does not accept integers for a 'number' in an alternate even
though it does for numbers outside of an alternate.

Mixing 'int' and 'number' in the same alternate is unusual,
since both are supplied by json-numbers, but there does not
seem to be a technical reason to forbid it given that our
json lexer distinguishes between json-numbers that can be
represented as an int vs. those that cannot.

Improve the existing test_visitor_in_alternate() to match the
style of the new test_visitor_in_alternate_number(), and to
ensure full coverage of all possible qtype parsing.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1443565276-4535-9-git-send-email-eblake@redhat.com>
[Eric's follow-up fixes squashed in]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-10-12 18:46:13 +02:00
Eric Blake
d220fbcd1d qapi: Test for various name collisions
Expose some weaknesses in the generator: we don't always forbid
the generation of structs that contain multiple members that map
to the same C or QMP name.  This has already been marked FIXME in
qapi.py in commit d90675f, but having more tests will make sure
future patches produce desired behavior; and updating existing
patches to better document things doesn't hurt, either.  Some of
these collisions are already caught in the old-style parser
checks, but ultimately we want all collisions to be caught in the
new-style QAPISchema*.check() methods.

This patch focuses on C struct members, and does not consider
collisions between commands and events (affecting C function
names), or even collisions between generated C type names with
user type names (for things like automatic FOOList struct
representing array types or FOOKind for an implicit enum).

There are two types of struct collisions we want to catch:
 1) Collision between two keys in a JSON object. qapi.py prevents
    that within a single struct (see test duplicate-key), but it is
    possible to have collisions between a type's members and its
    base type's members (existing tests struct-base-clash,
    struct-base-clash-deep), and its flat union variant members
    (renamed test flat-union-clash-member).
 2) Collision between two members of the C struct that is generated
    for a given QAPI type:
    a) Multiple QAPI names map to the same C name (new test
       args-name-clash)
    b) A QAPI name maps to a C name that is used for another purpose
       (new tests flat-union-clash-branch, struct-base-clash-base,
       union-clash-data). We already fixed some such cases in commit
       0f61af3e and 1e6c1616, but more remain.
    c) Two C names generated for other purposes clash
       (updated test alternate-clash, new test union-clash-branches,
       union-clash-type, flat-union-clash-type)

Ultimately, if we need to have a flat union where a tag value
clashes with a base member name, we could change the generator to
name the union (using 'foo.u.value' rather than 'foo.value') or
otherwise munge the C name corresponding to tag values.  But
unless such a need arises, it will probably be easier to just
forbid these collisions.

Some of these negative tests will be deleted later, and positive
tests added to qapi-schema-test.json in their place, when the
generator code is reworked to avoid particular code generation
collisions in class 2).

[Note that viewing this patch with git rename detection enabled
may see some confusion due to renaming some tests while adding
others, but where the content is similar enough that git picks
the wrong pre- and post-patch files to associate]

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1443565276-4535-6-git-send-email-eblake@redhat.com>
[Improve commit message and comments a bit, drop an unrelated test]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-10-12 18:44:54 +02:00
Markus Armbruster
28770e057f qapi: Introduce a first class 'any' type
It's first class, because unlike '**', it actually works, i.e. doesn't
require 'gen': false.

'**' will go away next.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2015-09-21 09:56:49 +02:00
Markus Armbruster
05f43a9608 qapi-event: Convert to QAPISchemaVisitor, fixing data with base
Fixes events whose data is struct with base to include the struct's
base members.  Test case is qapi-schema-test.json's event
__org.qemu_x-command:

    { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

    { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member2': 'str' } }

    { 'struct': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

Patch's effect on generated qapi_event_send___org_qemu_x_event():

    -void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
    +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
    +                                        const char *__org_qemu_x_member2,
                                             Error **errp)
     {
         QDict *qmp;
    @@ -224,6 +225,10 @@ void qapi_event_send___org_qemu_x_event(
             goto clean;
         }

    +    visit_type___org_qemu_x_Enum(v, &__org_qemu_x_member1, "__org.qemu_x-member1", &local_err);
    +    if (local_err) {
    +        goto clean;
    +    }
         visit_type_str(v, (char **)&__org_qemu_x_member2, "__org.qemu_x-member2", &local_err);
         if (local_err) {
             goto clean;

Code is generated in a different order now, but that doesn't matter.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2015-09-21 09:56:48 +02:00
Markus Armbruster
441cbac0c7 qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Fixes flat unions to visit the base's base members (the previous
commit merely added them to the struct).  Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

     static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
     {
         Error *err = NULL;

    +    visit_type_int(m, &(*obj)->integer, "integer", &err);
    +    if (err) {
    +        goto out;
    +    }
         visit_type_str(m, &(*obj)->string, "string", &err);
         if (err) {
             goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration
type.  None of them are currently used, obviously.  Example:
block-core.json's BlockdevRef now generates
visit_type_BlockdevRefKind().

Code is generated in a different order now, and therefore has got a
few new forward declarations.  Doesn't matter.

The guard QAPI_VISIT_BUILTIN_VISITOR_DECL is renamed to
QAPI_VISIT_BUILTIN.

The previous commit's two ugly special cases exist here, too.  Mark
both TODO.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2015-09-21 09:56:40 +02:00
Markus Armbruster
2b162ccbe8 qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Fixes flat unions to get the base's base members.  Test case is from
commit 2fc0043, in qapi-schema-test.json:

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

    { 'struct': 'UserDefUnionBase',
      'base': 'UserDefZero',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'struct': 'UserDefZero',
      'data': { 'integer': 'int' } }

Patch's effect on UserDefFlatUnion:

     struct UserDefFlatUnion {
         /* Members inherited from UserDefUnionBase: */
    +    int64_t integer;
         char *string;
         EnumOne enum1;
         /* Own members: */
         union { /* union tag is @enum1 */
             void *data;
             UserDefA *value1;
             UserDefB *value2;
             UserDefB *value3;
         };
     };

Flat union visitors remain broken.  They'll be fixed next.

Code is generated in a different order now, but that doesn't matter.

The two guards QAPI_TYPES_BUILTIN_STRUCT_DECL and
QAPI_TYPES_BUILTIN_CLEANUP_DECL are replaced by just
QAPI_TYPES_BUILTIN.

Two ugly special cases for simple unions now stand out like sore
thumbs:

1. The type tag is named 'type' everywhere, except in generated C,
   where it's 'kind'.

2. QAPISchema lowers simple unions to semantically equivalent flat
   unions.  However, the C generated for a simple unions differs from
   the C generated for its equivalent flat union, and we therefore
   need special code to preserve that pointless difference for now.

Mark both TODO.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2015-09-21 09:53:16 +02:00
Daniel P. Berrange
351d36e454 qapi: allow override of default enum prefix naming
The camel_to_upper() method applies some heuristics to turn
a mixed case type name into an all-uppercase name. This is
used for example, to generate enum constant name prefixes.

The heuristics don't also generate a satisfactory name
though. eg

  { 'enum': 'QCryptoTLSCredsEndpoint',
    'data': ['client', 'server']}

Results in Q_CRYPTOTLS_CREDS_ENDPOINT_CLIENT. This has
an undesirable _ after the initial Q and is missing an
_ between the CRYPTO & TLS strings.

Rather than try to add more and more heuristics to try
to cope with this, simply allow the QAPI schema to
specify the desired enum constant prefix explicitly.

eg

  { 'enum': 'QCryptoTLSCredsEndpoint',
    'prefix': 'QCRYPTO_TLS_CREDS_ENDPOINT',
    'data': ['client', 'server']}

Now gives the QCRYPTO_TLS_CREDS_ENDPOINT_CLIENT name.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
2015-09-15 10:59:28 +01:00
Markus Armbruster
80e60a19a8 tests/qapi-schema: Restore test case for flat union base bug
Test case added in commit 2fc0043, and messed up in commit 5223070.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2015-09-04 15:47:14 +02:00
Markus Armbruster
999387782f tests/qapi-schema: Document events with base don't work
When event FOO's 'data' is a struct with a base, we consider only the
struct's direct members, and ignore its base.  The generated
qapi_event_send_foo() doesn't take arguments for base members.

No such events currently exist in the QMP schema.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2015-09-04 15:47:14 +02:00
Markus Armbruster
422e16aac4 tests/qapi-schema: Document alternate's enum lacks visit function
We generate a declaration, but no definition.

The QMP schema has two: Qcow2OverlapChecks and BlockdevRef.  Neither
visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind()
is actually used.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2015-09-04 15:47:14 +02:00
Markus Armbruster
8c3f8e7721 qapi-visit: Fix generated code when schema has forward refs
The visit_type_implicit_FOO() are generated on demand, right before
their first use.  Used by visit_type_STRUCT_fields() when STRUCT has
base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat
union member, visit_type_implicit_FOO() calls
visit_type_implicit_FOO() before its definition, which doesn't
compile.

Rearrange qapi-schema-test.json to demonstrate the bug.

Fix by generating the necessary forward declaration.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2015-09-04 15:47:13 +02:00
Eric Blake
e3c4c3d796 qapi: Support downstream events and commands
Enhance the testsuite to cover downstream events and commands.
Events worked without more tweaks, but commands needed a few final
updates in the generator to mangle names in the appropriate places.
In making those tweaks, it was easier to drop type_visitor() and
inline its actions instead.

Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-14 18:21:27 +02:00
Eric Blake
d1f07c86c0 qapi: Support downstream alternates
Enhance the testsuite to cover downstream alternates, including
whether the branch name or type is downstream.  Update the
generator to mangle alternate names in the appropriate places.

Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-14 18:21:24 +02:00
Eric Blake
857af5f06c qapi: Support downstream flat unions
Enhance the testsuite to cover downstream flat unions, including
the base type, discriminator name and type, and branch name and
type.  Update the generator to mangle the union names in the
appropriate places.

Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-14 18:21:21 +02:00
Eric Blake
bb33729043 qapi: Support downstream simple unions
Enhance the testsuite to cover downstream simple unions, including
when a union branch is a downstream name.  Update the generator to
mangle the union names in the appropriate places.

Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-14 18:21:17 +02:00
Eric Blake
83a02706bb qapi: Support downstream structs
Enhance the testsuite to cover downstream structs, including struct
members and base structs.  Update the generator to mangle the
struct names in the appropriate places.

Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-14 18:21:13 +02:00
Eric Blake
fce384b8e5 qapi: Support downstream enums
Enhance the testsuite to cover a downstream enum type and enum
string.  Update the generator to mangle the enum name in the
appropriate places.

Signed-off-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-14 18:21:09 +02:00
Eric Blake
6446a59276 qapi: Drop tests for inline nested structs
A future patch will be using a 'name':{dictionary} entry in the
QAPI schema to specify a default value for an optional argument;
but existing use of inline nested structs conflicts with that goal.

More precisely, a definition in the QAPI schema associates a name
with a set of properties:

Example 1: { 'struct': 'Foo', 'data': { MEMBERS... } }
associates the global name 'Foo' with properties (meta-type struct)
and MEMBERS...

Example 2: 'mumble': TYPE
within MEMBERS... above associates 'mumble' with properties (type
TYPE) and (optional false) within type Foo

The syntax of example 1 is extensible; if we need another property,
we add another name/value pair to the dictionary (such as
'base':TYPE).  The syntax of example 2 is not extensible, because
the right hand side can only be a type.

We have used name encoding to add a property: "'*mumble': 'int'"
associates 'mumble' with (type int) and (optional true).  Nice,
but doesn't scale.  So the solution is to change our existing uses
to be syntactic sugar to an extensible form:

   NAME: TYPE   --> NAME:  { 'type': TYPE, 'optional': false }
   *ONAME: TYPE --> ONAME: { 'type': TYPE, 'optional': true }

This patch fixes the testsuite to avoid inline nested types, by
breaking the nesting into explicit types; it means that the type
is now boxed instead of unboxed in C code, but makes no difference
on the wire (and if desired, a later patch could change the
generator to not do so much boxing in C).  When touching code to
add new allocations, also convert existing allocations to
consistently prefer typesafe g_new0 over g_malloc0 when a type
name is involved.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:02 +02:00
Eric Blake
b6fcf32d9b qapi: Merge UserDefTwo and UserDefNested in tests
In the testsuite, UserDefTwo and UserDefNested were identical
structs other than the member names.  Reduce code duplication by
having just one type, and choose names that also favor reuse.
This will also make it easier for a later patch to get rid of
inline nested types in QAPI.  When touching code related to
allocations, convert g_malloc0(sizeof(Type)) to the more typesafe
g_new0(Type, 1).

Ensure that 'make check-qapi-schema check-unit' still passes.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:02 +02:00
Eric Blake
895a2a80e0 qapi: Use 'struct' instead of 'type' in schema
Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing.  Do the bulk of
the conversion to "struct" in qapi schema, with a fairly
mechanical:

for f in `find -name '*.json'; do sed -i "s/'type'/'struct'/"; done

followed by manually filtering out the places where we have a
'type' embedded in 'data'.  Then tweak a couple of tests whose
output changes slightly due to longer lines.

I also verified that the generated files for QMP and QGA (such
as qmp-commands.h) are the same before and after, as assurance
that I didn't leave in any accidental member name changes.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:01 +02:00
Eric Blake
ab916faddd qapi: Use 'alternate' to replace anonymous union
Previous patches have led up to the point where I create the
new meta-type "'alternate':'Foo'".  See the previous patches
for documentation; I intentionally split as much work into
earlier patches to minimize the size of this patch, but a lot
of it is churn due to testsuite fallout after updating to the
new type.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:00 +02:00
Eric Blake
ab04526744 qapi: Rename anonymous union type in test
Reduce churn in the future patch that replaces anonymous unions
with a new metatype 'alternate' by changing 'AnonUnion' to
'Alternate'.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:00 +02:00
Eric Blake
a8d4a2e4d7 qapi: Forbid base without discriminator in unions
None of the existing QMP or QGA interfaces uses a union with a
base type but no discriminator; it is easier to avoid this in the
generator to save room for other future extensions more likely to
be useful.  An earlier commit added a union-base-no-discriminator
test to ensure that we eventually give a decent error message;
likewise, removing UserDefUnion outright is okay, because we moved
all the tests we wish to keep into the tests of the simple union
UserDefNativeListUnion in the previous commit.  Now is the time to
actually forbid simple union with base, and remove the last
vestiges from the testsuite.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:00 +02:00
Eric Blake
cb17f79eef qapi: Fix generation of 'size' builtin type
We were missing the 'size' builtin type (which means that QAPI using
[ 'size' ] would fail to compile).

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
2015-05-05 18:39:00 +02:00