The QAPI code generator clocks in at some 3100 SLOC in 8 source files.
Almost 60% of the code is in qapi/common.py. Split it into more
focused modules:
* Move QAPISchemaPragma and QAPISourceInfo to qapi/source.py.
* Move QAPIError and its sub-classes to qapi/error.py.
* Move QAPISchemaParser and QAPIDoc to parser.py. Use the opportunity
to put QAPISchemaParser first.
* Move check_expr() & friends to qapi/expr.py. Use the opportunity to
put the code into a more sensible order.
* Move QAPISchema & friends to qapi/schema.py
* Move QAPIGen and its sub-classes, ifcontext,
QAPISchemaModularCVisitor, and QAPISchemaModularCVisitor to qapi/gen.py
* Delete camel_case(), it's unused since commit e98859a9b9 "qapi:
Clean up after recent conversions to QAPISchemaVisitor"
A number of helper functions remain in qapi/common.py. I considered
moving the code generator helpers to qapi/gen.py, but decided not to.
Perhaps we should rewrite them as methods of QAPIGen some day.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-7-armbru@redhat.com>
[Add "# -*- coding: utf-8 -*-" lines]
The next commit will split up qapi/common.py. gen_enum() needs
QAPISchemaEnumMember, and that's in the way. Move it to qapi/types.py
along with its buddy gen_enum_lookup().
Permit me a short a digression on history: how did gen_enum() end up
in qapi/common.py? Commit 21cd70dfc1 "qapi script: add event support"
duplicated qapi-types.py's gen_enum() and gen_enum_lookup() in
qapi-event.py. Simply importing them would have been cleaner, but
wasn't possible as qapi-types.py was a program, not a module. Commit
efd2eaa6c2 "qapi: De-duplicate enum code generation" de-duplicated by
moving them to qapi.py, which was a module.
Since then, program qapi-types.py has morphed into module types.py.
It's where gen_enum() and gen_enum_lookup() started, and where they
belong.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-6-armbru@redhat.com>
The frontend can't be run more than once due to its global state.
A future commit will want to do that.
The only global frontend state remaining is accidental:
QAPISchemaParser.__init__()'s parameter previously_included=[].
Python evaluates the default once, at definition time. Any
modifications to it are visible in subsequent calls. Well-known
Python trap. Change the default to None and replace it by the real
default in the function body. Use the opportunity to convert
previously_included to a set.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-4-armbru@redhat.com>
The frontend can't be run more than once due to its global state.
A future commit will want to do that.
Recent commit "qapi: Move context-sensitive checking to the proper
place" got rid of many global variables already, but pragma state is
still stored in global variables (that's why a pragma directive's
scope is the complete schema).
Move the pragma state to QAPISourceInfo.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-3-armbru@redhat.com>
Commit bc52d03ff5 "qapi: Make doc comments optional where we don't
need them" made scripts/qapi2texi.py fail[*] unless the schema had
pragma 'doc-required': true. The stated reason was inability to cope
with incomplete documentation.
When commit fb0bc835e5 "qapi-gen: New common driver for code and doc
generators" folded scripts/qapi2texi.py into scripts/qapi-gen.py, it
turned the failure into silent suppression.
The doc generator can cope with incomplete documentation now. I don't
know since when, or what the problem was, or even whether it ever
existed.
Drop the silent suppression.
[*] The fail part was broken, fixed in commit e8ba07ea9a.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-2-armbru@redhat.com>
qapi-gen.py crashes when it can't open the main schema file, and when
it can't read from any schema file. Lazy.
Change QAPISchema.__init__() to take a file name instead of a file
object. Move the open code from _include() to __init__(), so it's
used for the main schema file, too.
Move the read into the try for good measure, and rephrase the error
message.
Reporting open or read failure for the main schema file needs a
QAPISourceInfo representing "no source". Make QAPISourceInfo cope
with fname=None.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-27-armbru@redhat.com>
Point to the previous definition, unless it's a built-in.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-26-armbru@redhat.com>
Have check_exprs() check this later, so the error message gains an "in
definition line". Tweak the error message.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-25-armbru@redhat.com>
check_keys() has become a trivial wrapper for check_known_keys().
Eliminate it.
This makes its name available. Rename check_known_keys().
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-24-armbru@redhat.com>
check_if()'s errors don't point to the offending part of the
expression. For instance:
tests/qapi-schema/alternate-branch-if-invalid.json:2: 'if' condition ' ' makes no sense
Other check_FOO() do, with the help of a @source argument. Make
check_if() do that, too. The example above improves to:
tests/qapi-schema/alternate-branch-if-invalid.json:2: 'if' condition ' ' of 'data' member 'branch' makes no sense
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190927134639.4284-23-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Many error messages refer to the offending definition even though
they're preceded by an "in definition" line. Rephrase them.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190927134639.4284-22-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Have check_exprs() call check_keys() later, so its error messages gain
an "in definition" line.
Both check_keys() and check_name_is_str() check the definition's name
is a string. Since check_keys() now runs after check_name_is_str()
rather than before, its check is dead. Bury it. Checking values in
check_keys() is unclean anyway.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-21-armbru@redhat.com>
Split check_flags() off check_keys() and have check_exprs() call it
later, so its error messages gain an "in definition" line. Tweak the
error messages.
Checking values in a function named check_keys() is unclean anyway.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-20-armbru@redhat.com>
Move check_if() from check_keys() to check_exprs() and call it later,
so its error messages gain an "in definition" line.
Checking values in a function named check_keys() is unclean anyway.
The original sin was commit 0545f6b887 "qapi: Better error messages
for bad expressions", which checks the value of key 'name'. More
sinning in commit 2cbf09925a "qapi: More rigorous checking for type
safety bypass", commit c818408e44 "qapi: Implement boxed types for
commands/events", and commit 967c885108 "qapi: add 'if' to top-level
expressions". This commit does penance for the latter. The next
commits will do penance for the others.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-19-armbru@redhat.com>
QAPISchemaCommand.check() and QAPISchemaEvent().check() check 'data'
is present when 'boxed': true. That's context-free. Move to
check_command() and check_event().
Tweak the error message while there.
check_exprs() & friends now check exactly what qapi-code-gen.txt calls
the second layer of syntax.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-18-armbru@redhat.com>
When we introduced the QAPISchema intermediate representation (commit
ac88219a6c), we took a shortcut: we left check_exprs() & friends
alone instead of moving semantic checks into the
QAPISchemaFOO.check(). The .check() assert check_exprs() did its job.
Time to finish the conversion job. Move exactly the context-sensitive
checks to the .check(). They replace assertions there. Context-free
checks stay put.
Fixes the misleading optional tag error demonstrated by test
flat-union-optional-discriminator.
A few other error message improve.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-17-armbru@redhat.com>
check_name() consists of check_name_is_str() and check_name_str().
check_union() relies on the latter to catch optional discriminators.
The next commit will replace that by a more straightforward check.
Inlining check_name() into check_union() now should make that easier
to review.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-16-armbru@redhat.com>
Future commits will need info in the .check() methods of
QAPISchemaMember and its descendants. Get it there.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-15-armbru@redhat.com>
check_type() checks the array's contents, then peels off the array and
falls through to the "not array" code without resetting allow_array
and allow_dict to False. Works because the peeled value is a string,
and allow_array and allow_dict aren't used then. Tidy up anyway:
recurse instead, defaulting allow_array and allow_dict to False.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-14-armbru@redhat.com>
The checks for reserved names are spread far and wide. Move one from
add_name() to new check_defn_name_str(). This is a first step towards
collecting them all in dedicated name checking functions next to
check_name().
While there, drop the quotes around the meta-type in
check_name_str()'s error messages: "'command' uses ... name 'NAME'"
becomes "command uses ... name 'NAME'".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-13-armbru@redhat.com>
The special "does not allow optional name" error is well meant, but
confusing in practice. Drop it.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-12-armbru@redhat.com>
Replace check_name() by check_name_str() where the name is known to be
a string.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-11-armbru@redhat.com>
Split check_name() into check_name_is_str() and check_name_str(), keep
check_name() as a wrapper.
Move add_name()'s call into its caller check_exprs(), and inline.
This permits delaying check_name_str() there, so its error message
gains an "in definition" line.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-10-armbru@redhat.com>
Most check_FOO() take the thing being checked as first argument.
check_name(), check_type(), check_known_keys() don't. Clean that up.
While there, drop a "Todo" comment that should have been dropped in
commit 87adbbffd4 "qapi: add a dictionary form for TYPE".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-9-armbru@redhat.com>
We report name clashes like this:
struct-base-clash.json: In struct 'Sub':
struct-base-clash.json:5: 'name' (member of Sub) collides with 'name' (member of Base)
The "(member of Sub)" is redundant with "In struct 'Sub'". Comes from
QAPISchemaMember.describe(). Pass info to it, so it can detect the
redundancy and avoid it. Result:
struct-base-clash.json: In struct 'Sub':
struct-base-clash.json:5: member 'name' collides with member 'name' of type 'Base'
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-8-armbru@redhat.com>
Starting error messages with a capital letter complicates things when
text can get interpolated both at the beginning and in the middle of
an error message. The next patch will do that. Switch to lower case
to keep it simpler.
For what it's worth, the GNU Coding Standards advise the message
"should not begin with a capital letter when it follows a program name
and/or file name, because that isn’t the beginning of a sentence. (The
sentence conceptually starts at the beginning of the line.)"
While there, avoid breaking lines containing multiple arguments in the
middle of an argument.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-7-armbru@redhat.com>
QAPISchemaMember.check_clash() checks for member names that map to the
same c_name(). Takes care of rejecting duplicate names.
It also checks a naming rule: no uppercase in member names. That's a
rather odd place to do it. Enforcing naming rules is
check_name_str()'s job.
qapi-code-gen.txt specifies the name case rule applies to the name as
it appears in the schema. check_clash() checks c_name(name) instead.
No difference, as c_name() leaves alone case, but unclean.
Move the name case check into check_name_str(), less the c_name().
New argument @permit_upper suppresses it. Pass permit_upper=True for
definitions (which are not members), and when the member's owner is
whitelisted with pragma name-case-whitelist.
Bonus: name-case-whitelist now applies to a union's inline base, too.
Update qapi/qapi-schema.json pragma to whitelist union CpuInfo instead
of CpuInfo's implicit base type's name q_obj_CpuInfo-base.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-6-armbru@redhat.com>
We take pains to include the offending expression in error messages,
e.g.
tests/qapi-schema/alternate-any.json:2: alternate 'Alt' member 'one' cannot use type 'any'
But not always:
tests/qapi-schema/enum-if-invalid.json:2: 'if' condition must be a string or a list of strings
Instead of improving them one by one, report the offending expression
whenever it is known, like this:
tests/qapi-schema/enum-if-invalid.json: In enum 'TestIfEnum':
tests/qapi-schema/enum-if-invalid.json:2: 'if' condition must be a string or a list of strings
Error messages that mention the offending expression become a bit
redundant, e.g.
tests/qapi-schema/alternate-any.json: In alternate 'Alt':
tests/qapi-schema/alternate-any.json:2: alternate 'Alt' member 'one' cannot use type 'any'
I'll take care of that later in this series.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-5-armbru@redhat.com>
We track source locations with a dict of the form
{'file': FNAME, 'line': LINENO, 'parent': PARENT}
where PARENT is None for the main file, and the include directive's
source location for included files.
This is serviceable enough, but the next commit will add information,
and that's going to come out cleaner if we turn this into a class. So
do that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-4-armbru@redhat.com>
QAPISchemaMember.owner is the name of the defining entity. That's a
confusing name when an object type inherits members from a base type.
Rename it to .defined_in. Rename .set_owner() and ._pretty_owner() to
match.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-3-armbru@redhat.com>
When we introduced the QAPISchema intermediate representation (commit
ac88219a6c), we took a shortcut: we left check_exprs() & friends
alone instead of moving semantic checks into the
QAPISchemaFOO.check(). check_exprs() still checks and reports errors,
and the .check() assert check_exprs() did the job. There are a few
gaps, though.
QAPISchemaArrayType.check() neglects to assert the element type is not
an array. Add the assertion.
QAPISchemaObjectTypeVariants.check() neglects to assert the tag member
is not optional. Add the assertion.
It neglects to assert the tag member is not conditional. Add the
assertion.
It neglects to assert we actually have variants. Add the assertion.
It asserts the variants are object types, but neglects to assert they
don't have variants. Tighten the assertion.
QAPISchemaObjectTypeVariants.check_clash() has the same issue.
However, it can run only after .check(). Delete the assertion instead
of tightening it.
QAPISchemaAlternateType.check() neglects to assert the branch types
don't conflict. Fixing that isn't trivial, so add just a TODO comment
for now. It'll be resolved later in this series.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-2-armbru@redhat.com>
Easy since the previous commit provides .checked.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-20-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Entity checking goes back to commit ac88219a6c "qapi: New QAPISchema
intermediate representation", v2.5.0. It's designed to work as
follows: QAPISchema.check() calls .check() for all the schema's
entities. An entity's .check() recurses into another entity's
.check() only if the C struct generated for the former contains the C
struct generated for the latter (pointers don't count). This is used
to detect "object contains itself".
There are two instances of this:
* An object's C struct contains its base's C struct
QAPISchemaObjectType.check() calls self.base.check()
* An object's C struct contains its variants' C structs
QAPISchemaObjectTypeVariants().check calls v.type.check(). Since
commit b807a1e1e3 "qapi: Check for QAPI collisions involving variant
members", v2.6.0.
Thus, only object types can participate in recursion.
QAPISchemaObjectType.check() is made for that: it checks @self when
called the first time, recursing into base and variants, it reports an
"contains itself" error when this recursion reaches an object being
checked, and does nothing it reaches an object that has been checked
already.
The other .check() may safely assume they get called exactly once.
Sadly, this design has since eroded:
* QAPISchemaCommand.check() and QAPISchemaEvent.check() call
.args_type.check(). Since commit c818408e44 "qapi: Implement boxed
types for commands/events", v2.7.0. Harmless, since args_type can
only be an object type.
* QAPISchemaEntity.check() calls ._ifcond.check() when inheriting the
condition from another type. Since commit 4fca21c1b0 qapi: leave
the ifcond attribute undefined until check(), v3.0.0. This makes
simple union wrapper types recurse into the wrapped type (nothing
else uses this condition inheritance). The .check() of types used
as simple union branch type get called multiple times.
* QAPISchemaObjectType.check() calls its super type's .check()
*before* the conditional handling multiple calls. Also since commit
4fca21c1b0. QAPISchemaObjectType.check()'s guard against multiple
checking doesn't protect QAPISchemaEntity.check().
* QAPISchemaArrayType.check() calls .element_type.check(). Also since
commit 4fca21c1b0. The .check() of types used as array element
types get called multiple times.
Commit 56a4689582 "qapi: Fix array first used in a different module"
(v4.0.0) added more code relying on this .element_type.check().
The absence of explosions suggests the .check() involved happen to be
effectively idempotent.
Fix the unwanted recursion anyway:
* QAPISchemaCommand.check() and QAPISchemaEvent.check() calling
.args_type.check() is unnecessary. Delete the calls.
* Fix QAPISchemaObjectType.check() to call its super type's .check()
after the conditional handling multiple calls.
* A QAPISchemaEntity's .ifcond becomes valid at .check(). This is due
to arrays and simple unions.
Most types get ifcond and info passed to their constructor.
Array types don't: they get it from their element type, which
becomes known only in .element_type.check().
The implicit wrapper object types for simple union branches don't:
they get it from the wrapped type, which might be an array.
Ditch the idea to set .ifcond in .check(). Instead, turn it into a
property and compute it on demand. Safe because it's only used
after the schema has been checked.
Most types simply return the ifcond passed to their constructor.
Array types forward to their .element_type instead, and the wrapper
types forward to the wrapped type.
* A QAPISchemaEntity's .module becomes valid at .check(). This is
because computing it needs info and schema.fname, and because array
types get it from their element type instead.
Make it a property just like .ifcond.
Additionally, have QAPISchemaEntity.check() assert it gets called at
most once, so the design invariant will stick this time. Neglecting
that was entirely my fault.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-19-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[Commit message tidied up]
QAPISchemaObjectType.check() does nothing for types that have been
checked already. Except the "has been checked" predicate is broken
for empty types: self.members is [] then, which isn't true. The bug
is harmless, but fix it anyway: use self.member is not None instead.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-18-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Commit bceae7697f "qapi script: support enum type as discriminator in
union" made check_exprs() add the implicit enum types of simple unions
to global @enum_types. I'm not sure it was needed even then. It's
certainly not needed now. Delete it.
discriminator_find_enum_define() and add_name() parameter @implicit
are now dead. Bury them.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-17-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
All callers pass a dict argument to @keys, except check_keys() passes
a dict's .keys(). Drop .keys() there, and rename parameter @keys to
@value.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-16-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
check_keys() parameter expr_elem expects a dictionary with keys 'expr'
and 'info'. Passing the two values separately is simpler, so do that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-15-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
We normalize shorthand to longhand forms in check_expr(): enumeration
values with normalize_enum(), feature values with
normalize_features(), struct members, union branches and alternate
branches with normalize_members(). If conditions are an exception: we
normalize them in QAPISchemaEntity.check() and
QAPISchemaMember.__init(), with listify_cond(). The idea goes back to
commit 2cbc94376e "qapi: pass 'if' condition into QAPISchemaEntity
objects", v3.0.0.
Normalize in check_expr() instead, with new helper normalize_if().
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-14-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Commit 87adbbffd4..3e270dcacc "qapi: Add 'if' to (implicit
struct|union|alternate) members" (v4.0.0) neglected test coverage, and
promptly failed to check the conditions. Review fail.
Recent commit "tests/qapi-schema: Demonstrate insufficient 'if'
checking" added test coverage, demonstrating the bug. Fix it by add
the missing check_if().
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-13-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
"'if': 'COND'" generates "#if COND". We reject empty COND because it
won't compile. Blank COND won't compile any better, so reject that,
too.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-12-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
check_union() checks the discriminator exists in base and makes sense.
Two error messages mention the base. These are broken for anonymous
bases, as demonstrated by tests flat-union-invalid-discriminator and
flat-union-invalid-if-discriminator.err. The third one doesn't
bother.
First broken when commit ac4338f8eb "qapi: Allow anonymous base for
flat union" (v2.6.0) neglected to adjust the "not a member of base"
error message. Commit ccadd6bcba "qapi: Add 'if' to implicit struct
members" (v4.0.0) then cloned the flawed error message.
Dumb them down not to mention the base.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-11-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
We represent the parse tree as OrderedDict. We fetch optional dict
members with .get(). So far, so good.
We represent null literals as None. .get() returns None both for
"absent" and for "present, value is the null literal". Uh-oh.
Test features-if-invalid exposes this bug: "'if': null" is
misinterpreted as absent "if".
We added null to the schema language to "allow [...] an explicit
default value" (commit e53188ada5 "qapi: Allow true, false and null in
schema json", v2.4.0). Hasn't happened; null is still unused except
as generic invalid value in tests/.
To fix, we'd have to replace .get() by something more careful, or
represent null differently. Feasible, but we got more and bigger fish
to fry right now. Remove the null literal from the schema language.
Replace null in tests by another invalid value.
Test features-if-invalid now behaves as it should.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-10-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Show text up to next structural character, whitespace, or quote
character instead of just the first character.
Forgotten quotes now get reported like "Stray 'command'" instead of
"Stray 'c'".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-9-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Consistently enclose error messages in double quotes. Use single
quotes within, except for one case of "'".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190914153506.2151-8-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
The previous commit made qapi-code-gen.txt define "(top-level)
expression" as either "directive" or "definition". The code still
uses "expression" when it really means "definition". Tidy up.
The previous commit made qapi-code-gen.txt use "object" rather than
"dictionary". The code still uses "dictionary". Tidy up.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-17-armbru@redhat.com>
For consistency with docs/devel/qapi-code-gen.txt.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-12-armbru@redhat.com>
Absent flat union branches default to the empty struct (since commit
800877bb16 "qapi: allow empty branches in flat unions"). But an
attempt to omit all of them is rejected with "Union 'FOO' has no
branches". Harmless oddity, but it's easy to avoid, so do that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-11-armbru@redhat.com>
[Commit message typo fixed]
A union or alternate without branches makes no sense and doesn't work:
it can't be instantiated. A union or alternate with just one branch
works, but is degenerate. We accept the former, but reject the
latter. Weird. docs/devel/qapi-code-gen.txt doesn't mention the
difference. It claims an alternate definition is "is similar to a
simple union type".
Permit degenerate alternates to make them consistent with unions.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-10-armbru@redhat.com>
We reject empty types with 'boxed': true. We don't really need that
to work, but making it work is actually simpler than rejecting it, so
do that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-9-armbru@redhat.com>
Since the previous commit restricted strings to printable ASCII,
\uXXXX's only use is obfuscation. Drop it.
This leaves \\, \/, \', and \". Since QAPI schema strings are all
names, and names are restricted to ASCII letters, digits, hyphen, and
underscore, none of them is useful.
The latter three have no test coverage. Drop them.
Keep \\ to avoid (more) gratuitous incompatibility with JSON.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-8-armbru@redhat.com>
RFC 8259 on string contents:
All Unicode characters may be placed within the quotation marks,
except for the characters that MUST be escaped: quotation mark,
reverse solidus, and the control characters (U+0000 through
U+001F).
The QAPI schema parser accepts both less and more than JSON: it
accepts only ASCII with \u (less), and accepts control characters
other than LF (new line) unescaped. How it treats unescaped non-ASCII
input differs between Python 2 and Python 3.
Make it accept strictly less: require printable ASCII. Drop support
for \b, \f, \n, \r, \t.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-7-armbru@redhat.com>
Commands and events can define their argument type inline (default) or
by referring to another type ('boxed': true, since commit c818408e44
"qapi: Implement boxed types for commands/events", v2.7.0). The
unboxed inline definition is an (anonymous) struct type. The boxed
type may be a struct, union, or alternate type.
The latter is problematic: docs/interop/qemu-spec.txt requires the
value of the 'data' key to be a json-object, but any non-degenerate
alternate type has at least one branch that isn't.
Fortunately, we haven't made use of alternates in this context outside
tests/. Drop support for them.
QAPISchemaAlternateType.is_empty() is now unused. Drop it, too.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-4-armbru@redhat.com>
check_type() uses @allow_optional only when @value is a dictionary and
@allow_dict is True. All callers that pass allow_dict=True also pass
allow_optional=True.
Therefore, @allow_optional is always True when check_type() uses it.
Drop the redundant parameter.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-3-armbru@redhat.com>
QAPIDoc uses a state machine to for processing of documentation lines.
Its state is encoded as an enum QAPIDoc._state (well, as enum-like
class actually, thanks to our infatuation with Python 2).
All we ever do with the state is calling the state's function to
process a line of documentation. The enum values effectively serve as
handles for the functions.
Eliminate the rather wordy indirection: store the function to call in
QAPIDoc._append_line. Update and improve comments.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190606153803.5278-8-armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
[Commit message typo fixed]
Features will be documented in a new part introduced by a "Features:"
line, after arguments and before named sections.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20190606153803.5278-6-armbru@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Documentation comments follow a certain structure: First, we have a text
with a general description (called QAPIDoc.body). After this,
descriptions of the arguments follow. Finally, we have a part that
contains various named sections.
The code doesn't show this structure, but just checks various attributes
that indicate indirectly which part is being processed, so it happens to
do the right set of things in the right phase. This is hard to follow,
and adding support for documentation of features would be even harder.
This patch restructures the code so that the three parts are clearly
separated. The code becomes a bit longer, but easier to follow. The
resulting output remains unchanged.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20190606153803.5278-5-armbru@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Sometimes, the behaviour of QEMU changes without a change in the QMP
syntax (usually by allowing values or operations that previously
resulted in an error). QMP clients may still need to know whether
they can rely on the changed behavior.
Let's add feature flags to the QAPI schema language, so that we can make
such changes visible with schema introspection.
An example for a schema definition using feature flags looks like this:
{ 'struct': 'TestType',
'data': { 'number': 'int' },
'features': [ 'allow-negative-numbers' ] }
Introspection information then looks like this:
{ "name": "TestType", "meta-type": "object",
"members": [
{ "name": "number", "type": "int" } ],
"features": [ "allow-negative-numbers" ] }
This patch implements feature flags only for struct types. We'll
implement them more widely as needed.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20190606153803.5278-2-armbru@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
No header includes qemu-common.h after this commit, as prescribed by
qemu-common.h's file comment.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190523143508.25387-5-armbru@redhat.com>
[Rebased with conflicts resolved automatically, except for
include/hw/arm/xlnx-zynqmp.h hw/arm/nrf51_soc.c hw/arm/msf2-soc.c
block/qcow2-refcount.c block/qcow2-cluster.c block/qcow2-cache.c
target/arm/cpu.h target/lm32/cpu.h target/m68k/cpu.h target/mips/cpu.h
target/moxie/cpu.h target/nios2/cpu.h target/openrisc/cpu.h
target/riscv/cpu.h target/tilegx/cpu.h target/tricore/cpu.h
target/unicore32/cpu.h target/xtensa/cpu.h; bsd-user/main.c and
net/tap-bsd.c fixed up]
We generally put implicitly defined types in whatever module triggered
their definition. This is wrong for array types, as the included test
case demonstrates. Let's have a closer look at it.
Type 'Status' is defined sub-sub-module.json. Array type ['Status']
occurs in main module qapi-schema-test.json and in
include/sub-module.json. The main module's use is first, so the array
type gets put into the main module.
The generated C headers define StatusList in qapi-types.h. But
include/qapi-types-sub-module.h uses it without including
qapi-types.h. Oops.
To fix that, put the array type into its element type's module.
Now StatusList gets generated into qapi-types-sub-module.h, which all
its users include.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190301154051.23317-8-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
The #include directives to pull in sub-modules use file names relative
to the main module. Works only when all modules are in the same
directory, or the main module's output directory is in the compiler's
include path. Use relative file names instead.
The dummy variable we generate to avoid empty .o files has an invalid
name for sub-modules in other directories. Fix that.
Both messed up in commit 252dc3105f "qapi: Generate separate .h, .c
for each module". Escaped testing because tests/qapi-schema-test.json
doesn't cover sub-modules in other directories, only
tests/qapi-schema/include-relpath.json does, and we generate and
compile C code only for the former, not the latter. Fold the latter
into the former. This would have caught the mistakes fixed in this
commit.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190301154051.23317-5-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Not much of an improvement now, but the next commit will profit.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190301154051.23317-4-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
This reverts commit 7bd2634905.
The commit applied the events' conditions to the members of enum
QAPIEvent. Awkward, because it renders QAPIEvent unusable in
target-independent code as soon as we make an event target-dependent.
Reverting this has the following effects:
* ui/vnc.c can remain target independent.
* monitor_qapi_event_conf[] doesn't have to muck around with #ifdef.
* query-events again doesn't reflect conditionals. I'm going to
deprecate it in favor of query-qmp-schema.
Another option would be to split target-dependent parts off enum
QAPIEvent into a target-dependent enum. Doesn't seem worthwhile right
now.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190214152251.2073-17-armbru@redhat.com>
Having to include qapi-events.h just for QAPIEvent is suboptimal, but
quite tolerable now. It'll become problematic when we have events
conditional on the target, because then qapi-events.h won't be usable
from target-independent code anymore. Avoid that by generating it
into separate files.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190214152251.2073-6-armbru@redhat.com>
The next commit wants to generate qapi-emit-events.{c.h}. To enable
that, extend QAPISchemaModularCVisitor to support additional "system
modules", i.e. modules that don't correspond to a (user-defined) QAPI
schema module.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190214152251.2073-5-armbru@redhat.com>
We neglect to call .visit_module() for the special module we use for
built-ins. Harmless, but clean it up anyway. The
tests/qapi-schema/*.out now show the built-in module as 'module None'.
Subclasses of QAPISchemaModularCVisitor need to ._add_module() this
special module to enable code generation for built-ins. When this
hasn't been done, QAPISchemaModularCVisitor.visit_module() does
nothing for the special module. That looks like built-ins could
accidentally be generated into the wrong module when a subclass
neglects to call ._add_module(). Can't happen, because built-ins are
all visited before any other module. But that's non-obvious. Switch
off code generation explicitly.
Rename QAPISchemaModularCVisitor._begin_module() to
._begin_user_module().
New QAPISchemaModularCVisitor._is_builtin_module(), for clarity.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190214152251.2073-4-armbru@redhat.com>
The qapi_event_send_FOO() functions emit events like this:
QMPEventFuncEmit emit;
emit = qmp_event_get_func_emit();
if (!emit) {
return;
}
qmp = qmp_event_build_dict("FOO");
[put event arguments into @qmp...]
emit(QAPI_EVENT_FOO, qmp);
The value of qmp_event_get_func_emit() depends only on the program:
* In qemu-system-FOO, it's always monitor_qapi_event_queue.
* In tests/test-qmp-event, it's always event_test_emit.
* In all other programs, it's always null.
This is exactly the kind of dependence the linker is supposed to
resolve; we don't actually need an indirection.
Note that things would fall apart if we linked more than one QAPI
schema into a single program: each set of qapi_event_send_FOO() uses
its own event enumeration, yet they share a single emit function.
Which takes the event enumeration as an argument. Which one if
there's more than one?
More seriously: how does this work even now? qemu-system-FOO wants
QAPIEvent, and passes a function taking that to
qmp_event_set_func_emit(). test-qmp-event wants test_QAPIEvent, and
passes a function taking that to qmp_event_set_func_emit().
It works by type trickery, of course:
typedef void (*QMPEventFuncEmit)(unsigned event, QDict *dict);
void qmp_event_set_func_emit(QMPEventFuncEmit emit);
QMPEventFuncEmit qmp_event_get_func_emit(void);
We use unsigned instead of the enumeration type. Relies on both
enumerations boiling down to unsigned, which happens to be true for
the compilers we use.
Clean this up as follows:
* Generate qapi_event_send_FOO() that call PREFIX_qapi_event_emit()
instead of the value of qmp_event_set_func_emit().
* Generate a prototype for PREFIX_qapi_event_emit() into
qapi-events.h.
* PREFIX_ is empty for qapi/qapi-schema.json, and test_ for
tests/qapi-schema/qapi-schema-test.json. It's qga_ for
qga/qapi-schema.json, and doc-good- for
tests/qapi-schema/doc-good.json, but those don't define any events.
* Rename monitor_qapi_event_queue() to qapi_event_emit() instead of
passing it to qmp_event_set_func_emit(). This takes care of
qemu-system-FOO.
* Rename event_test_emit() to test_qapi_event_emit() instead of
passing it to qmp_event_set_func_emit(). This takes care of
tests/test-qmp-event.
* Add a qapi_event_emit() that does nothing to stubs/monitor.c. This
takes care of all other programs that link code emitting QMP events.
* Drop qmp_event_set_func_emit(), qmp_event_get_func_emit().
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181218182234.28876-3-armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
[Commit message typos fixed]
Default branches variant should use the member conditional.
This fixes compilation with --disable-replication.
Fixes: 335d10cd8e
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20181217204046.14861-1-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Long line wrapped]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Use a common function to generate the "If:..." line.
While at it, get rid of the existing \n\n (no idea why it was
there). Use a line-break in member description, this seems to look
slightly better in the plaintext version.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20181213123724.4866-19-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Wrap generated enum and struct members and their supporting code with
#if/#endif, using the .ifcond members added in the previous patches.
We do enum and struct in a single patch because union tag enum and the
associated variants tie them together, and dealing with that to split
the patch doesn't seem worthwhile.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-18-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The generated code is for now *unconditional*. Later patches generate
the conditionals.
Note that union discriminators may not have 'if' conditionals.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-14-marcandre.lureau@redhat.com>
Message-Id: <20181213123724.4866-15-marcandre.lureau@redhat.com>
[Patches squashed, commit message tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Wherever a struct/union/alternate/command/event member with NAME: TYPE
form is accepted, desugar it to a NAME: { 'type': TYPE } form.
This will allow to add new member details, such as 'if' in the
following patch to introduce conditionals, or 'default' for default
values etc.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20181213123724.4866-13-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Add condition to QAPIEvent enum members based on the event 'if'.
The generated code remains unconditional for now. Later patches
generate the conditionals (also there is no additional coverage of
this change in qapi-schema-test.out since the event_names enum is an
implicit type created by qapi/events.py).
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-11-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
QAPISchemaMember gains .ifcond for enum members: inherited classes,
such as QAPISchemaObjectTypeMember, will thus have an ifcond member
after this (those different types will also use the .ifcond to store
the condition and generate conditional code in the following patches).
The generated code remains unconditional for now. Later patches
generate the conditionals.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-10-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Desugar the enum NAME form to { 'name': NAME }. This will allow to add
new enum members, such as 'if' in the following patch.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20181213123724.4866-7-marcandre.lureau@redhat.com>
Message-Id: <20181213123724.4866-8-marcandre.lureau@redhat.com>
Message-Id: <20181213123724.4866-9-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Harmless accidental move backed out, long line wrapped, patches
squashed]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Report the set of missing or unknown keys. And give a hint about the
accepted keys.
The error message for multiple meta type members (visible in
tests/qapi-schema/double-type.err) is not improved.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-6-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Introduce a new helper function to check if the given keys are known,
and if mandatory keys are present. The function will be reused in
other places in the following code changes.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-5-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
This will allow to add and access more properties associated with enum
values/members, like the associated 'if' condition. We may want to
have a specialized type QAPISchemaEnumMember, for now this will do.
Modify gen_enum() and gen_enum_lookup() for the same reason.
Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-3-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The generated C enumeration types explicitly set the enumeration
constants to 0, 1, 2, ... That's exactly what you get when you don't
supply values.
Drop the explicit values. No change now, but it will avoid gaps in
the values when we later add support for 'if' conditions. Avoiding
such gaps will save us the trouble of changing the ENUM_lookup[]
tables to work without a sentinel.
We'll have to take care to ensure the headers required by the 'if'
conditions get always included before the generated QAPI code.
Fortunately, our convention to include "qemu/osdep.h" first in any .c
ensures that's the case for our CONFIG_FOO macros.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181213123724.4866-2-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Rename QAPISchemaEnumType.values and related variables to members.
Makes sense ever since commit 93bda4dd4 changed .values from list of
string to list of QAPISchemaMember. Obvious no-op.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20181208111606.8505-4-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We consciously chose in commit 1a9a507b to hide QAPI type names
from the introspection output on the wire, but added a command
line option -u to unmask the type name when doing a debug build.
The unmask option still remains useful to some other forms of
automated analysis, so it will not be removed; however, when it
is not in use, the generated .c file can be hard to read. At
the time when we first introduced masking, the generated file
consisted only of a monolithic C string, so there was no clean
way to inject any comments.
Later, in commit 7d0f982b, we switched the generation to output
a QLit object, in part to make it easier for future addition of
conditional compilation. In fact, commit d626b6c1 took advantage
of this by passing a tuple instead of a bare object for encoding
the output of conditionals. By extending that tuple, we can now
interject strategic comments.
For now, type name debug aid comments are only output once per
meta-type, rather than at all uses of the number used to encode
the type within the introspection data. But this is still a lot
more convenient than having to regenerate the file with the
unmask operation temporarily turned on - merely search the
generated file for '"NNN" =' to learn the corresponding source
name and associated definition of type NNN.
The generated qapi-introspect.c changes only with the addition
of comments, such as:
| @@ -14755,6 +15240,7 @@
| { "name", QLIT_QSTR("[485]"), },
| {}
| })),
| + /* "485" = QCryptoBlockInfoLUKSSlot */
| QLIT_QDICT(((QLitDictEntry[]) {
| { "members", QLIT_QLIST(((QLitObject[]) {
| QLIT_QDICT(((QLitDictEntry[]) {
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180827213943.33524-3-eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Rebased, update to qapi-code-gen.txt corrected]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Commit 7d0f982b changed generated introspection output to no longer
produce long lines in the generated .c file, but failed to adjust
comments to match. Add some clarity that the shorter length that
matters most is the overall QMP response on the wire.
Commit 25b1ef31 triggers a pep8 formatting nit.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180827213943.33524-2-eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We emit a dummy variable in each .c file "to shut up OSX toolchain
warnings about empty .o files" (commit 252dc3105f). Separate it from
the code preceding it (if any) with a blank line.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180828120736.32323-2-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
The generated qapi_event_send_FOO() take an Error ** argument. They
can't actually fail, because all they do with the argument is passing it
to functions that can't fail: the QObject output visitor, and the
@qmp_emit callback, which is either monitor_qapi_event_queue() or
event_test_emit().
Drop the argument, and pass &error_abort to the QObject output visitor
and @qmp_emit instead.
Suggested-by: Eric Blake <eblake@redhat.com>
Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180815133747.25032-4-peterx@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten, update to qapi-code-gen.txt corrected]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
build_params() returns '' instead of 'void' when there are no
parameters. Can't happen now, but the next commit will change that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
[peterx: compose the patch from email replies]
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180815133747.25032-3-peterx@redhat.com>
Fix the following issues:
common.py:873:13: E129 visually indented line with same indent as next logical line
common.py:1766:5: E741 ambiguous variable name 'l'
common.py:1784:1: E305 expected 2 blank lines after class or function definition, found 1
common.py:1833:1: E305 expected 2 blank lines after class or function definition, found 1
common.py:1843:1: E305 expected 2 blank lines after class or function definition, found 1
visit.py:181:18: E127 continuation line over-indented for visual indent
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20180621083551.775-1-armbru@redhat.com>
[Fixup squashed in:]
Message-ID: <871sd0nzw9.fsf@dusky.pond.sub.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Making 'allow-oob' optional in SchemaInfoCommand permits omitting it
in the common case. Shrinks query-qmp-schema's output from 122.1KiB
to 118.6KiB for me.
Note that out-of-band execution is still experimental (you have to
configure the monitor with x-oob=on to use it).
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180718090557.17248-1-armbru@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
According to commit 047f7038f5, option --preconfig
[...] allows pausing QEMU in the new RUN_STATE_PRECONFIG state,
allowing the configuration of QEMU from QMP before the machine
jumps into board initialization code of machine_run_board_init()
The intent is to allow management to query machine state and
additionally configure it using previous query results within one
QEMU instance (i.e. eliminate the need to start QEMU twice, 1st to
query board specific parameters and 2nd for actual VM start using
query results for additional parameters).
The implementation is a bit of a hack: it splices in an additional
main loop before machine creation, in special runstate preconfig. New
command exit-preconfig exits that main loop. QEMU continues
initializing, creates the machine, and runs the good old main loop.
The replacement of the main loop is transparent to monitors.
Sadly, some commands expect initialization to be complete. Running
them in --preconfig's main loop violates their preconditions. Since
we don't really know which commands are safe, we use a whitelist.
This drags the concept of run state into the QMP core.
The whitelist is done as a command flag in the QAPI schema (commit
d6fe3d02e9). Drags the concept of run state further into the QAPI
language.
The command flag is exposed in query-qmp-schema (also commit
d6fe3d02e9). This makes it ABI.
I consider the whole thing an offensively ugly hack, but sometimes an
ugly hack is the best we can do to solve a problem people have.
The need described by the commit message quote above is genuine. The
proper solution would be a main loop that permits complete
configuration via QMP. This is out of reach, thus the hack.
However, even though the need is genuine, it isn't urgent: libvirt is
not going to use this anytime soon. Baking a hack into ABI before it
has any users is a bad idea.
This commit reverts the parts of commit d6fe3d02e9 that affect ABI
via query-qmp-schema. The commit did the following:
(1) Add command flag 'allow-preconfig' to the QAPI schema language
(2) Pass it to code generators
(3) Have the commands.py code generator pass it to the command
registry (so commit 047f7038f5 can use it as whitelist)
(4) Add 'allow-preconfig' to SchemaInfoCommand (neglecting to update
qapi-code-gen.txt section "Client JSON Protocol introspection")
(5) Set 'allow-preconfig': true for commands qmp_capabilities,
query-commands, query-command-line-options, query-status
Revert exactly (4), plus a bit of documentation added to
qemu-tech.info in commit 047f7038f5.
Shrinks query-qmp-schema's output from 126.5KiB to 121.8KiB for me.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180705091402.26244-2-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Acked-by: Eduardo Habkost <ehabkost@redhat.com>
Acked-by: Igor Mammedov <imammedo@redhat.com>
[Straightforward conflict with commit d626b6c1ae resolved]
The documentation is generated only once, and doesn't know C
pre-conditions. Add 'If:' sections for top-level entities.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-13-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Types & visitors are coupled and must be handled together to avoid
temporary build regression.
Wrap generated types/visitor code with #if/#endif using the context
helpers. Derived from a patch by Marc-André.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-12-marcandre.lureau@redhat.com>
Wrap generated code with #if/#endif using an 'ifcontext' on
QAPIGenCSnippet objects.
This makes a conditional event's qapi_event_send_FOO() compile-time
conditional, but its enum QAPIEvent member remains unconditional for
now. A follow up patch "qapi-event: add 'if' condition to implicit
event enum" will improve this.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-11-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Wrap generated code with #if/#endif using an 'ifcontext' on
QAPIGenCSnippet objects.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180703155648.11933-10-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Line breaks tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
This commit adds 'ifcond' conditions to top-level QLit objects.
Future work will add them to object and enum type members, i.e. within
QLit objects.
Extend the QLit generator to_qlit() to accept (@obj, @cond) tuples in
addition to just @obj. The tuple causes the QLit generated for
objects for @obj with #if/#endif conditions for @cond.
See generated tests/test-qmp-introspect.c. Example diff after this
patch:
--- before 2018-01-08 11:55:24.757083654 +0100
+++ tests/test-qmp-introspect.c 2018-01-08 13:08:44.477641629 +0100
@@ -51,6 +51,8 @@
{ "name", QLIT_QSTR("EVENT_F"), },
{}
})),
+#if defined(TEST_IF_CMD)
+#if defined(TEST_IF_STRUCT)
QLIT_QDICT(((QLitDictEntry[]) {
{ "arg-type", QLIT_QSTR("5"), },
{ "meta-type", QLIT_QSTR("command"), },
@@ -58,12 +60,16 @@
{ "ret-type", QLIT_QSTR("0"), },
{}
})),
+#endif /* defined(TEST_IF_STRUCT) */
+#endif /* defined(TEST_IF_CMD) */
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-9-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The following patch is going to break list entries with #if/#endif, so
they should have the trailing ',' as suffix.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-8-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Add helpers to wrap generated code with #if/#endif lines.
A later patch wants to use QAPIGen for generating C snippets rather
than full C files with copyright headers etc. Splice in class
QAPIGenCCode between QAPIGen and QAPIGenC.
Add a 'with' statement context manager that will be used to wrap
generator visitor methods. The manager will check if code was
generated before adding #if/#endif lines on QAPIGenCSnippet
objects. Used in the following patches.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180703155648.11933-7-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Skip preprocessor lines when adding indentation, since that would
likely result in invalid code.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-6-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Modify the test visitor to check correct passing of values.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-5-marcandre.lureau@redhat.com>
[Accidental change to roms/seabios dropped]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We commonly initialize attributes to None in .init(), then set their
real value in .check(). Accessing the attribute before .check()
yields None. If we're lucky, the code that accesses the attribute
prematurely chokes on None.
It won't for .ifcond, because None is a legitimate value.
Leave the ifcond attribute undefined until check().
Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-4-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Built-in objects remain unconditional. Explicitly defined objects use
the condition specified in the schema. Implicitly defined objects
inherit their condition from their users. For most of them, there is
exactly one user, so the condition to use is obvious. The exception
is wrapped types generated for simple union variants, which can be
shared by any number of simple unions. The tight condition would be
the disjunction of the conditions of these simple unions. For now,
use the wrapped type's condition instead. Much simpler and good
enough for now.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-3-marcandre.lureau@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Accept 'if' key in top-level elements, accepted as string or list of
string type. The following patches will modify the test visitor to
check the value is correctly saved, and generate #if/#endif code (as a
single #if/endif line or a series for a list).
Example of 'if' key:
{ 'struct': 'TestIfStruct', 'data': { 'foo': 'int' },
'if': 'defined(TEST_IF_STRUCT)' }
The generated code is for now *unconditional*. Later patches generate
the conditionals.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180703155648.11933-2-marcandre.lureau@redhat.com>
[Commit message and Documentation improved]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Commit 1a9a507b2e "qapi-introspect: Hide type names" added local
variable @jsons to improve sorting, but also removed the sorting. It
was part of a big series that went to v8, and it made sense until v2
or so...
Commit 7d0f982bfb replaced @jsons by @qlits, preserving the
uselessness.
Get rid of it.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180620124742.16979-1-armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Python 2 happily reads UTF-8 files in text mode, but Python 3 requires
either UTF-8 locale or an explicit encoding passed to open(). Commit
d4e5ec877c fixed this by setting the en_US.UTF-8 locale. Falls apart
when the locale isn't be available.
Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use
binary mode instead, with manual conversion from bytes to str. Works,
but opening with an explicit encoding is simpler, so do that.
Since Python 2's open() doesn't support the encoding parameter, we
need to suppress it with a version check.
Reported-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
Reported-by: Matthias Maier <tamiko@43-1.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180618175958.29073-2-armbru@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
It often happens that just a few discriminator values imply extra data in
a flat union. Existing checks did not make possible to leave other values
uncovered. Such cases had to be worked around by either stating a dummy
(empty) type or introducing another (subset) discriminator enumeration.
Both options create redundant entities in qapi files for little profit.
With this patch it is not necessary anymore to add designated union
fields for every possible value of a discriminator enumeration.
Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
Message-Id: <1529311206-76847-2-git-send-email-anton.nefedov@virtuozzo.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The event generator produces an enum, and put it in the last visited
module. It fits better in the main module, since it's the set of all
visited events, from all modules.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180321115211.17937-3-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Python 2.7 (the minimum Python version we require) provides
collections.OrderedDict on the standard library, so we don't need
to carry our own implementation.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20180608175252.25110-3-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
New option will be used to allow commands, which are prepared/need
to run, during preconfig state. Other commands that should be able
to run in preconfig state, should be amended to not expect machine
in initialized state or deal with it.
For compatibility reasons, commands that don't use new flag
'allow-preconfig' explicitly are not permitted to run in
preconfig state but allowed in all other states like they used
to be.
Within this patch allow following commands in preconfig state:
qmp_capabilities
query-qmp-schema
query-commands
query-command-line-options
query-status
exit-preconfig
to allow qmp connection, basic introspection and moving to the next
state.
PS:
set-numa-node and query-hotpluggable-cpus will be enabled later in
a separate patches.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Message-Id: <1526057503-39287-1-git-send-email-imammedo@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[ehabkost: Changed "since 2.13" to "since 3.0"]
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
We'll soon need an enumeration type that lists all the softmmu targets
that QEMU (the project) supports. Introduce @SysEmuTarget to
"common.json".
The enum constant @x86_64 doesn't match the QAPI convention of preferring
hyphen ("-") over underscore ("_"). This is intentional; the @SysEmuTarget
constants are supposed to produce QEMU executable names when stringified
and appended to the "qemu-system-" prefix. Put differently, the
replacement text of the TARGET_NAME preprocessor macro must be possible to
look up in the list of (stringified) enum constants.
Like other enum types, @SysEmuTarget too can be used for discriminator
fields in unions. For the @i386 constant, a C-language union member called
"i386" would be generated. On mingw build hosts, "i386" is a macro
however. Add "i386" to "polluted_words" at once.
Cc: "Daniel P. Berrange" <berrange@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20180427192852.15013-3-lersek@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Now that we can safely call QOBJECT() on QObject * as well as its
subtypes, we can have macros qobject_ref() / qobject_unref() that work
everywhere instead of having to use QINCREF() / QDECREF() for QObject
and qobject_incref() / qobject_decref() for its subtypes.
The replacement is mechanical, except I broke a long line, and added a
cast in monitor_qmp_cleanup_req_queue_locked(). Unlike
qobject_decref(), qobject_unref() doesn't accept void *.
Note that the new macros evaluate their argument exactly once, thus no
need to shout them.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180419150145.24795-4-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Rebased, semantic conflict resolved, commit message improved]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
It was missed in the first version of OOB series. We should check this
to make sure we throw the right error when fault value is passed in.
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180326063901.27425-5-peterx@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Here "oob" stands for "Out-Of-Band". When "allow-oob" is set, it means
the command allows out-of-band execution.
The "oob" idea is proposed by Markus Armbruster in following thread:
https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html
This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
well for command entries, so that QMP clients can know which commands
can be used in out-of-band calls. For example the command "migrate"
originally looks like:
{"name": "migrate", "ret-type": "17", "meta-type": "command",
"arg-type": "86"}
And it'll be changed into:
{"name": "migrate", "ret-type": "17", "allow-oob": false,
"meta-type": "command", "arg-type": "86"}
This patch only provides the QMP interface level changes. It does not
contain the real out-of-band execution implementation yet.
Suggested-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180309090006.10018-18-peterx@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: rebase on introspection done by qlit]
Signed-off-by: Eric Blake <eblake@redhat.com>
Replace the generated json string with a literal qobject. The later is
easier to deal with, at run time as well as compile time: adding #if
conditionals will be easier than in a json string.
The output of query-qmp-schema is not changed.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180305172951.2150-5-marcandre.lureau@redhat.com>
[eblake: fix python 3 failure]
Signed-off-by: Eric Blake <eblake@redhat.com>
Move qapi-schema.json to qapi/, so it's next to its modules, and all
files get generated to qapi/, not just the ones generated for modules.
Consistently name the generated files qapi-MODULE.EXT:
qmp-commands.[ch] become qapi-commands.[ch], qapi-event.[ch] become
qapi-events.[ch], and qmp-introspect.[ch] become qapi-introspect.[ch].
This gets rid of the temporary hacks in scripts/qapi/commands.py,
scripts/qapi/events.py, and scripts/qapi/common.py.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-28-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: Fix trailing dot in tpm.c, undo temporary hack for OSX toolchain]
Signed-off-by: Eric Blake <eblake@redhat.com>
In my "build everything" tree, a change to the types in
qapi-schema.json triggers a recompile of about 4800 out of 5100
objects.
The previous commit split up qmp-commands.h, qmp-event.h, qmp-visit.h,
qapi-types.h. Each of these headers still includes all its shards.
Reduce compile time by including just the shards we actually need.
To illustrate the benefits: adding a type to qapi/migration.json now
recompiles some 2300 instead of 4800 objects. The next commit will
improve it further.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-24-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
[eblake: rebase to master]
Signed-off-by: Eric Blake <eblake@redhat.com>
Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth. These monolithic headers get included all
over the place. In my "build everything" tree, adding a QAPI type
recompiles about 4800 out of 5100 objects.
We wouldn't write such monolithic headers by hand. It stands to
reason that we shouldn't generate them, either.
Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header
qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.
Mirror the schema's includes in the headers, so that qapi-types.h gets
you everything exactly as before. If you need less, you can include
one or more of the sub-module headers. To be exploited shortly.
Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h,
qmp-commands.c, qapi-event.h, qapi-event.c the same way.
qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.
The split of qmp-commands.c duplicates static helper function
qmp_marshal_output_str() in qapi-commands-char.c and
qapi-commands-misc.c. This happens when commands returning the same
type occur in multiple modules. Not worth avoiding.
Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and
qmp-commands.[ch] to qapi-commands.[ch], name the shards that way
already, to reduce churn. This requires temporary hacks in
commands.py and events.py. Similarly, c_name() must temporarily
be taught to munge '/' in common.py. They'll go away with the rename.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-23-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: declare a dummy variable in each .c file, to shut up OSX
toolchain warnings about empty .o files, including hacking c_name()]
Signed-off-by: Eric Blake <eblake@redhat.com>
guardname() fails to return a valid C identifier for arguments
containing anything but [A-Za-z0-9_.-']. Fix that. Don't bother
protecting ticklish identifiers; header guards are all-caps, and no
ticklish identifiers are.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180211093607.27351-22-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Linking code from multiple separate QAPI schemata into the same
program is possible, but involves some weirdness around built-in
types:
* We generate code for built-in types into .c only with option
--builtins. The user is responsible for generating code for exactly
one QAPI schema per program with --builtins.
* We generate code for built-in types into .h regardless of
--builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all
copies of this code are exactly the same, including any combination
of these headers works.
Replace this contraption by something more conventional: generate code
for built-in types into their very own files: qapi-builtin-types.c,
qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but
only with --builtins. Obey --output-dir, but ignore --prefix for
them.
Make qapi-types.h include qapi-builtin-types.h. With multiple
schemata you now have multiple qapi-types.[ch], but only one
qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and
qapi-builtin-visit.[ch].
Bonus: if all you need is built-in stuff, you can include a much
smaller header. To be exploited shortly.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-21-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: fix octal constant for python 3]
Signed-off-by: Eric Blake <eblake@redhat.com>
The use of QAPIGen is rather shallow so far: most of the output
accumulation is not converted. Take the next step: convert output
accumulation in the code-generating visitor classes. Helper functions
outside these classes are not converted.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-20-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: rebase to earlier guardstart cleanup]
Signed-off-by: Eric Blake <eblake@redhat.com>
All generated .c are named like their .h, except for qmp-marshal.c and
qmp-commands.h. To add to the confusion, tests-qmp-commands.c falsely
matches generated test-qmp-commands.h.
Get rid of this unnecessary complication.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-19-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
The include directive permits modular QAPI schemata, but the generated
code is monolithic all the same. To permit generating modular code,
the front end needs to pass more information on inclusions to the back
ends. The commit before last added the necessary information to the
parse tree. This commit adds it to the intermediate representation
and its QAPISchemaVisitor. A later commit will use this to to
generate modular code.
New entity QAPISchemaInclude represents inclusions. Call new visitor
method visit_include() for it, so visitors can see the sub-modules a
module includes.
Note that unlike other entities, QAPISchemaInclude has no name, and is
therefore not added to entity_dict.
New QAPISchemaEntity attribute @module names the entity's source file.
Call new visitor method visit_module() when it changes during a visit,
so visitors can keep track of the module being visited.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-18-armbru@redhat.com>
[eblake: avoid accidental deletion of self._predefining]
Signed-off-by: Eric Blake <eblake@redhat.com>
The generators' conversion to visitors (merge commit 9e72681d16)
changed the processing order of entities from source order to
alphabetical order. The next commit needs source order, so change it
back.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-17-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
The parse tree is a list of expressions. Except include expressions
currently get replaced by the included file's parse tree.
Instead of throwing away the include expression, keep it with the file
name expanded so you don't have to track the including file's
directory to make sense of it.
A future commit will put this include expression to use.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-16-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: fix check of expr after assignment]
Signed-off-by: Eric Blake <eblake@redhat.com>
Error messages print absolute file names of included files even if the
user gave a relative one on the command line:
$ PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json
Improve this to
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json
The error message when an include file can't be opened prints the
include directive's file name, which is relative to the including
file. Change this to print the file name relative to the working
directory. Visible in tests/qapi-schema/include-no-file.err.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-12-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
A massive number of objects depends on QAPI-generated headers. In my
"build everything" tree, it's roughly 4800 out of 5100. This is
particularly annoying when only some of the generated files change,
say for a doc fix.
Improve qapi-gen.py to touch its output files only if they actually
change. Rebuild time for a QAPI doc fix drops from many minutes to a
few seconds. Rebuilds get faster for certain code changes, too. For
instance, adding a simple QMP event now recompiles less than 200
instead of 4800 objects. But adding a QAPI type is as bad as ever;
we've clearly got more work to do.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180211093607.27351-11-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: fix octal constant for python3]
Signed-off-by: Eric Blake <eblake@redhat.com>
argparse is nicer to use than getopt, and gives us --help almost for
free.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-10-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
[eblake: Fix --output-dir editing accident]
Signed-off-by: Eric Blake <eblake@redhat.com>
Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files. Similar for qga/qapi-schema.json. This is
silly. Replace the six programs by a single program that spits out
all eleven files.
The programs become modules in new Python package qapi, along with the
helper library. This requires moving them to scripts/qapi/. While
moving them, consistently drop executable mode bits.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-9-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: move change to one-line 'blurb' earlier in series, mention mode
bit change as intentional, update qapi-code-gen.txt to match actual
generated events.c file]
Signed-off-by: Eric Blake <eblake@redhat.com>
