Commit Graph

1796 Commits

Author SHA1 Message Date
Volker Rümelin
c6e93c9d62 simpletrace: build() missing 2 required positional arguments
Commit 4e66c9ef64 "tracetool: add input filename and line number to
Event" forgot to add a line number and a filename argument at one
build method call site.

Traceback (most recent call last):
  File "./scripts/simpletrace.py", line 261, in <module>
    run(Formatter())
  File "./scripts/simpletrace.py", line 236, in run
    process(events, sys.argv[2], analyzer, read_header=read_header)
  File "./scripts/simpletrace.py", line 177, in process
    dropped_event =
      Event.build("Dropped_Event(uint64_t num_events_dropped)")
TypeError: build() missing 2 required positional arguments:
  'lineno' and 'filename'

Add the missing arguments.

Signed-off-by: Volker Rümelin <vr_qemu@t-online.de>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-id: 20210131173415.3392-1-vr_qemu@t-online.de
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:51:00 +00:00
Stefan Hajnoczi
418ed14268 trace: make the 'log' backend timestamp configurable
Timestamps in tracing output can be distracting. Make it possible to
control tid/timestamp printing with -msg timestamp=on|off. The default
is no tid/timestamps. Previously they were always printed.

Suggested-by: BALATON Zoltan <balaton@eik.bme.hu>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: BALATON Zoltan <balaton@eik.bme.hu>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210125113507.224287-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:50:55 +00:00
Daniel P. Berrangé
09612de7e9 tracetool: also strip %l and %ll from systemtap format strings
All variables are 64-bit and so %l / %ll are not required, and the
latter is actually invalid:

  $ sudo stap -e 'probe begin{printf ("BEGIN")}'  -I .
  parse error: invalid or missing conversion specifier
          saw: operator ',' at ./qemu-system-x86_64-log.stp:15118:101
       source:     printf("%d@%d vhost_vdpa_set_log_base dev: %p base: 0x%x size: %llu
refcnt: %d fd: %d log: %p\n", pid(), gettimeofday_ns(), dev, base, size, refcnt, fd, log)

                       ^

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Laurent Vivier <lvivier@redhat.com>
Message-id: 20210106130239.1004729-1-berrange@redhat.com

[Fixed "simiarly" typo found by Laurent Vivier <lvivier@redhat.com>
--Stefan]

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:50:55 +00:00
Laurent Vivier
9f4e519fd7 tracetool: fix "PRI" macro decoding
macro is not reset after use, so the format decoded is always the
one of the first "PRI" in the format string.

For instance:

  vhost_vdpa_set_config(void *dev, uint32_t offset, uint32_t size, \
                        uint32_t flags) "dev: %p offset: %"PRIu32" \
                        size: %"PRIu32" flags: 0x%"PRIx32

generates:

  printf("%d@%d vhost_vdpa_set_config dev: %p offset: %u size: %u \
          flags: 0x%u\n", pid(), gettimeofday_ns(), dev, offset, \
          size, flags)

for the "flags" parameter, we can see a "0x%u" rather than a "0x%x"
because the first macro was "PRIu32" (for offset).

In the loop, macro becomes "PRIu32PRIu32PRIx32", and c_macro_to_format()
returns always macro[3] ('u' in this case). This patch resets macro after
the format has been decoded.

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20210105191721.120463-3-lvivier@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:50:54 +00:00
Peter Maydell
74208cd252 * Replace --enable/disable-git-update with --with-git-submodules
to allow improved control over use of git submodules
 * Deprecate the -enable-fips option
 * Ensure docs use prefer format for bool options
 * Clarify platform support rules
 * Misc fixes to keymap conversions
 * Fix misc problems on macOS
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE2vOm/bJrYpEtDo4/vobrtBUQT98FAmAUQXUACgkQvobrtBUQ
 T9/9Zw//VV3z7MI70HjWk17TUmkKLGFbmrc3xvtS6EMZbhfXpsR8qMfSjoRxP4Gn
 CgeftOa/lK5XjL7FcEUmcZeO8jZNE/+S5wgP2upP5Ae1zuDeyaSPTGR6UYs7WqSa
 P1kQsot2sAsZCP7Lko0jSv7rEDInK1bcdWr0a/xR5M2TvKiXoEdbElIZdwe3yLbC
 qWPWiv2pp6z2eGtJK/9VBxXjP0tlkNAR+jz8p/8RSZEEKf3aSq6HvVae+bHU86nX
 9t4baZk3ZuuwkmHJrHJP/72BjjitqEOiydtSsOCVYmY2GOclYeTnezG/vm75cGIK
 ej6hwFGElAEQk+KU9g7HOXi1eqXncSBjP0xWqc8PaZLW7wW/Uv8UV94fk+SCkIQ0
 GVBBLP8PzySL+w33NN7sv14cWeBpsjy9arMr4okZ80klFlS2hx/FqnUmhFhoN8ub
 ptIPUcSJoL+6pjxatV2yJ6bHkVga9PiwtLC8/rqETYPJz0b756xlPaEGd+0B1P0V
 ZMraE7lCgS0kJM6w0xF3R7JlGVtKouj4ZePCK5MGjpByBAK/vn/MbhbEmcyh+uJD
 r6sakKwqLjDLKdmQDd+biq3q6mvKE8ZKaUj8dXYdvxXSxxqXZ8DnrNpj/UkDTi3S
 chlTT1qUYrp2MVYqGmFjZ0XcEdEUR1I39rUs+wRxE3RgtE56M1A=
 =g6bC
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/berrange-gitlab/tags/misc-fixes-pull-request' into staging

* Replace --enable/disable-git-update with --with-git-submodules
  to allow improved control over use of git submodules
* Deprecate the -enable-fips option
* Ensure docs use prefer format for bool options
* Clarify platform support rules
* Misc fixes to keymap conversions
* Fix misc problems on macOS

# gpg: Signature made Fri 29 Jan 2021 17:10:13 GMT
# gpg:                using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg:                 aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E  8E3F BE86 EBB4 1510 4FDF

* remotes/berrange-gitlab/tags/misc-fixes-pull-request:
  tests: Replace deprecated ASN1 code
  tests: Fix runtime error in test-authz-pam
  ui: update keycodemapdb submodule commit
  crypto: Add spaces around operator
  configure: replace --enable/disable-git-update with --with-git-submodules
  docs: fix missing backslash in certtool shell example
  docs: simplify and clarify the platform support rules
  Prefer 'on' | 'off' over 'yes' | 'no' for bool options
  os: deprecate the -enable-fips option and QEMU's FIPS enforcement
  crypto: Fix memory leaks in set_loaded for tls-*
  crypto: Forbid broken unloading of secrets
  crypto: Move USER_CREATABLE to secret_common base class
  crypto: Fix some code style problems, add spaces around operator

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-01-29 19:51:25 +00:00
Dan Streetman
7d7dbf9dc1 configure: replace --enable/disable-git-update with --with-git-submodules
Replace the --enable-git-update and --disable-git-update configure params
with the param --with-git-submodules=(update|validate|ignore) to
allow 3 options for building from a git repo.

This is needed because downstream packagers, e.g. Debian, Ubuntu, etc,
also keep the source code in git, but do not want to enable the
'git_update' mode; with the current code, that's not possible even
if the downstream package specifies --disable-git-update.

The previous parameters are deprecated but still available; the
--enable-git-update parameter maps to --with-git-submodules=update and
--disable-git-update parameter maps to --with-git-submodules=validate.

The configure script behavior is slightly modified, where previously
the dtc, capstone, and slirp submodules were not validated when
--disable-git-update was specified (but were updated with git-update
enabled), now they are validated when using --with-git-submodules=validate
and are only ignored when using --with-git-submodules=ignore.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-29 17:07:53 +00:00
Alexander Graf
8a74ce618b hvf: Add hypervisor entitlement to output binaries
In macOS 11, QEMU only gets access to Hypervisor.framework if it has the
respective entitlement. Add an entitlement template and automatically self
sign and apply the entitlement in the build.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Roman Bolshakov <r.bolshakov@yadro.com>
Tested-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-01-29 10:47:28 +00:00
Vladimir Sementsov-Ogievskiy
c701f59253 simplebench: add bench-backup.py
Add script to benchmark new backup architecture.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210116214705.822267-24-vsementsov@virtuozzo.com>
[mreitz: s/not unsupported/not supported/]
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
b2fcb0c575 simplebench: bench_block_job: add cmd_options argument
Add argument to allow additional block-job options.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-23-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-Ogievskiy
2096de521e simplebench/bench_block_job: use correct shebang line with python3
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-22-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Eric Farman
ab5ec23f9c update-linux-headers: Include const.h
Kernel commit a85cbe6159ff ("uapi: move constants from
<linux/kernel.h> to <linux/const.h>") breaks our script
because of the unrecognized include. Let's add that to
our processing.

Signed-off-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210104202057.48048-2-farman@linux.ibm.com>
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
2021-01-21 11:19:45 +01:00
Alex Bennée
9e5acb373d scripts/checkpatch.pl: fix git-show invocation to include diffstat
Without this checkpatch keeps complaining about new/changed files even
when MAINTAINERS has been updated. Normal invocations of checkpatch on
patch files rather than commit IDs are unaffected.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210114165730.31607-13-alex.bennee@linaro.org>
2021-01-18 10:04:31 +00:00
Philippe Mathieu-Daudé
4cacecaaa2 decodetree: Open files with encoding='utf-8'
When decodetree.py was added in commit 568ae7efae, QEMU was
using Python 2 which happily reads UTF-8 files in text mode.
Python 3 requires either UTF-8 locale or an explicit encoding
passed to open(). Now that Python 3 is required, explicit
UTF-8 encoding for decodetree source files.

To avoid further problems with the user locale, also explicit
UTF-8 encoding for the generated C files.

Explicit both input/output are plain text by using the 't' mode.

This fixes:

  $ /usr/bin/python3 scripts/decodetree.py test.decode
  Traceback (most recent call last):
    File "scripts/decodetree.py", line 1397, in <module>
      main()
    File "scripts/decodetree.py", line 1308, in main
      parse_file(f, toppat)
    File "scripts/decodetree.py", line 994, in parse_file
      for line in f:
    File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 80:
  ordinal not in range(128)

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Suggested-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20210110000240.761122-1-f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2021-01-13 08:39:08 -10:00
Maxim Levitsky
b9a0de3773 scripts/gdb: implement 'qemu bt'
This script first runs the regular gdb's 'bt' command, and then if we are in a
coroutine it prints the coroutines backtraces in the order in which they
were called.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201217155436.927320-3-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-01-12 12:38:03 +01:00
Maxim Levitsky
4cbf8efc5b scripts/gdb: fix 'qemu coroutine' when users selects a non topmost stack frame
The code that dumps the stack frame works like that:
* save current registers
* overwrite current registers (including rip/rsp) with coroutine snapshot
  in the jmpbuf
* print backtrace
* restore the saved registers.

If the user has currently selected a non topmost stack frame in gdb,
the above code will still restore the selected frame registers,
but the gdb will then lose the selected frame index, which makes it impossible
to switch back to frame 0, to continue debugging the executable.

Therefore switch temporarily to the topmost frame of the stack
for the above code.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201217155436.927320-2-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-01-12 12:38:03 +01:00
Qiuhao Li
4cc5752303 fuzz: heuristic split write based on past IOs
If previous write commands write the same length of data with the same step,
we view it as a hint.

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB3502480AD07811A6A49B8FEAFCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao Li
dd21ed0edf fuzz: add minimization options
-M1: remove IO commands iteratively
-M2: try setting bits in operand of write/out to zero

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350204C52E7A39E6B0EEC870FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao Li
9d20f2af53 fuzz: set bits in operand of write/out to zero
Simplifying the crash cases by opportunistically setting bits in operands of
out/write to zero may help to debug, since usually bit one means turn on or
trigger a function while zero is the default turn-off setting.

Tested bug https://bugs.launchpad.net/qemu/+bug/1908062

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB3502C84B6346A3E3DE708C7BFCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao Li
247ab240c2 fuzz: remove IO commands iteratively
Now we use a one-time scan and remove strategy in the minimizer,
which is not suitable for timing dependent instructions.

For example, instruction A will indicate an address where the config
chunk locates, and instruction B will make the configuration active.
If we have the following instruction sequence:

...
A1
B1
A2
B2
...

A2 and B2 are the actual instructions that trigger the bug.

If we scan from top to bottom, after we remove A1, the behavior of B1
might be unknowable, including not to crash the program. But we will
successfully remove B1 later cause A2 and B2 will crash the process
anyway:

...
A1
A2
B2
...

Now one more trimming will remove A1.

In the perfect case, we would need to be able to remove A and B (or C!) at
the same time. But for now, let's just add a loop around the minimizer.

Since we only remove instructions, this iterative algorithm is converging.

Tested with Bug 1908062.

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350263004448040ACCB9A9F1FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao Li
e72203abec fuzz: split write operand using binary approach
Currently, we split the write commands' data from the middle. If it does not
work, try to move the pivot left by one byte and retry until there is no
space.

But, this method has two flaws:

1. It may fail to trim all unnecessary bytes on the right side.

For example, there is an IO write command:

  write addr uuxxxxuu

u is the unnecessary byte for the crash. Unlike ram write commands, in most
case, a split IO write won't trigger the same crash, So if we split from the
middle, we will get:

  write addr uu (will be removed in next round)
  write addr xxxxuu

For xxxxuu, since split it from the middle and retry to the leftmost byte
won't get the same crash, we will be stopped from removing the last two
bytes.

2. The algorithm complexity is O(n) since we move the pivot byte by byte.

To solve the first issue, we can try a symmetrical position on the right if
we fail on the left. As for the second issue, instead moving by one byte, we
can approach the boundary exponentially, achieving O(log(n)).

Give an example:

                   xxxxuu len=6
                        +
                        |
                        +
                 xxx,xuu 6/2=3 fail
                        +
         +--------------+-------------+
         |                            |
         +                            +
  xx,xxuu 6/2^2=1 fail         xxxxu,u 6-1=5 success
                                 +   +
         +------------------+----+   |
         |                  |        +-------------+ u removed
         +                  +
   xx,xxu 5/2=2 fail  xxxx,u 6-2=4 success
                           +
                           |
                           +-----------+ u removed

In some rare cases, this algorithm will fail to trim all unnecessary bytes:

  xxxxxxxxxuxxxxxx
  xxxxxxxx-xuxxxxxx Fail
  xxxx-xxxxxuxxxxxx Fail
  xxxxxxxxxuxx-xxxx Fail
  ...

I think the trade-off is worth it.

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB3502D26F1BEB680CBBC169E5FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao Li
7b339f287f fuzz: double the IOs to remove for every loop
Instead of removing IO instructions one by one, we can try deleting multiple
instructions at once. According to the locality of reference, we double the
number of instructions to remove for the next round and recover it to one
once we fail.

This patch is usually significant for large input.

Test with quadrupled trace input at:
  https://bugs.launchpad.net/qemu/+bug/1890333/comments/1

Patched 1/6 version:
  real  0m45.904s
  user  0m16.874s
  sys   0m10.042s

Refined version:
  real  0m11.412s
  user  0m6.888s
  sys   0m3.325s

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350280A67BB55C3FADF173E3FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao Li
22ec0c696f fuzz: accelerate non-crash detection
We spend much time waiting for the timeout program during the minimization
process until it passes a time limit. This patch hacks the CLOSED (indicates
the redirection file closed) notification in QTest's output if it doesn't
crash.

Test with quadrupled trace input at:
  https://bugs.launchpad.net/qemu/+bug/1890333/comments/1

Original version:
  real	1m37.246s
  user	0m13.069s
  sys	0m8.399s

Refined version:
  real	0m45.904s
  user	0m16.874s
  sys	0m10.042s

Note:

Sometimes the mutated or the same trace may trigger a different crash
summary (second-to-last line) but indicates the same bug. For example, Bug
1910826 [1], which will trigger a stack overflow, may output summaries
like:

SUMMARY: AddressSanitizer: stack-overflow
/home/qiuhao/hack/qemu/build/../softmmu/physmem.c:488 in
flatview_do_translate

or

SUMMARY: AddressSanitizer: stack-overflow
(/home/qiuhao/hack/qemu/build/qemu-system-i386+0x27ca049) in __asan_memcpy

Etc.

If we use the whole summary line as the token, we may be prevented from
further minimization. So in this patch, we only use the first three words
which indicate the type of crash:

SUMMARY: AddressSanitizer: stack-overflow

[1] https://bugs.launchpad.net/qemu/+bug/1910826

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350251DC04003450348FAF68FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Peter Maydell
01b3e68bb1 scripts/coccinelle: New script to remove unnecessary timer_del() calls
Now that timer_free() implicitly calls timer_del(), sequences
  timer_del(mytimer);
  timer_free(mytimer);

can be simplified to just
  timer_free(mytimer);

Add a Coccinelle script to do this transformation.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20201215154107.3255-3-peter.maydell@linaro.org
2021-01-08 15:13:38 +00:00
Stefan Hajnoczi
7fb48c0ee1 tracetool: show trace-events filename/lineno in fmt string errors
The compiler encounters trace event format strings in generated code.
Format strings are error-prone and therefore clear compiler errors are
important.

Use the #line directive to show the trace-events filename and line
number in format string errors:
https://gcc.gnu.org/onlinedocs/gcc-10.2.0/cpp/Line-Control.html

For example, if the cpu_in trace event's %u is changed to %p the
following error is reported:

  trace-events:29:18: error: format ‘%p’ expects argument of type ‘void *’, but argument 7 has type ‘unsigned int’ [-Werror=format=]

Line 29 in trace-events is where cpu_in is defined. This works for any
trace-events file in the QEMU source tree and the correct path is
displayed.

Unfortunately there does not seem to be a way to set the column, so "18"
is not the right character on that line.

Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-5-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00
Stefan Hajnoczi
4e66c9ef64 tracetool: add input filename and line number to Event
Store the input filename and line number in Event.

A later patch will use this to improve error messages.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-4-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00
Stefan Hajnoczi
294170c1dd tracetool: add out_lineno and out_next_lineno to out()
Make the output file line number and next line number available to
out().

A later patch will use this to improve error messages.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-3-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00
Stefan Hajnoczi
c05012a365 tracetool: add output filename command-line argument
The tracetool.py script writes to stdout. This means the output filename
is not available to the script. Add the output filename to the
command-line so that the script has access to the filename.

This also simplifies the tracetool.py invocation. It's no longer
necessary to use meson's custom_build(capture : true) to save output.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-2-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00
Vladimir Sementsov-Ogievskiy
cff6d3ca43 scripts/simplebench: add bench_prealloc.py
Benchmark for new preallocate filter.

Example usage:
    ./bench_prealloc.py ../../build/qemu-img \
        ssd-ext4:/path/to/mount/point \
        ssd-xfs:/path2 hdd-ext4:/path3 hdd-xfs:/path4

The benchmark shows performance improvement (or degradation) when use
new preallocate filter with qcow2 image.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-22-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
181f60c8c7 simplebench/results_to_text: make executable
Make results_to_text a tool to dump results saved in JSON file.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-21-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
aa362403f4 simplebench/results_to_text: add difference line to the table
Performance improvements / degradations are usually discussed in
percentage. Let's make the script calculate it for us.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-20-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
[mreitz: 'seconds' instead of 'secs']
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
96be1aeec7 simplebench/results_to_text: improve view of the table
Move to generic format for floats and percentage for error.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-19-vsementsov@virtuozzo.com>
Acked-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
8e979febb0 simplebench: move results_to_text() into separate file
Let's keep view part in separate: this way it's better to improve it in
the following commits.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-18-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
bfccfa62ac simplebench: rename ascii() to results_to_text()
Next patch will use utf8 plus-minus symbol, let's use more generic (and
more readable) name.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-17-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
f52e1af0b0 scripts/simplebench: use standard deviation for +- error
Standard deviation is more usual to see after +- than current maximum
of deviations.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-16-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
4a44554a65 scripts/simplebench: support iops
Support benchmarks returning not seconds but iops. We'll use it for
further new test.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-15-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Vladimir Sementsov-Ogievskiy
270124e7ef scripts/simplebench: fix grammar: s/successed/succeeded/
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-14-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-12-18 12:35:55 +01:00
Philippe Mathieu-Daudé
a1bcbb485c scripts/git.orderfile: Keep files with .inc extension sorted
Sort .inc files along with the extension including them.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20201213205132.243628-1-f4bug@amsat.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-15 12:53:16 -05:00
Marc-André Lureau
07b35a23c3 compiler.h: remove QEMU_GNUC_PREREQ
When needed, the G_GNUC_CHECK_VERSION() glib macro can be used instead.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20201210134752.780923-14-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-15 12:53:15 -05:00
Paolo Bonzini
953d0c333e scripts: kernel-doc: remove unnecessary change wrt Linux
A comment in kernel-doc mentions QEMU's qatomic_set macro, but since
this code originated in Linux we should just revert it and stay as close
to the kernel's copy of the script as possible.

The change was introduced (more or less unintentionally) in QEMU commit
commit d73415a315, which did a global search-and-replace of QEMU's
atomic access macros.

Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:25 -05:00
Mauro Carvalho Chehab
a1b8a57a0e scripts: kernel-doc: use :c:union when needed
Sphinx C domain code after 3.2.1 will start complaning if :c:struct
would be used for an union type:

	.../Documentation/gpu/drm-kms-helpers:352: ../drivers/video/hdmi.c:851: WARNING: C 'identifier' cross-reference uses wrong tag: reference name is 'union hdmi_infoframe' but found name is 'struct hdmi_infoframe'. Full reference name is 'union hdmi_infoframe'. Full found name is 'struct hdmi_infoframe'.

So, let's address this issue too in advance, in order to
avoid future issues.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Link: https://lore.kernel.org/r/6e4ec3eec914df62389a299797a3880ae4490f35.1603791716.git.mchehab+huawei@kernel.org
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-30-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:25 -05:00
Mauro Carvalho Chehab
e495a1b26c scripts: kernel-doc: split typedef complex regex
The typedef regex for function prototypes are very complex.
Split them into 3 separate regex and then join them using
qr.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Link: https://lore.kernel.org/r/3a4af999a0d62d4ab9dfae1cdefdfcad93383356.1603792384.git.mchehab+huawei@kernel.org
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-29-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:24 -05:00
Mauro Carvalho Chehab
01a7917d9b scripts: kernel-doc: fix typedef parsing
The include/linux/genalloc.h file defined this typedef:

	typedef unsigned long (*genpool_algo_t)(unsigned long *map,unsigned long size,unsigned long start,unsigned int nr,void *data, struct gen_pool *pool, unsigned long start_addr);

Because it has a type composite of two words (unsigned long),
the parser gets the typedef name wrong:

.. c:macro:: long

   **Typedef**: Allocation callback function type definition

Fix the regex in order to accept composite types when
defining a typedef for a function pointer.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Link: https://lore.kernel.org/r/328e8018041cc44f7a1684e57f8d111230761c4f.1603792384.git.mchehab+huawei@kernel.org
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-28-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:24 -05:00
Paolo Bonzini
3e72dc0135 Revert "kernel-doc: Handle function typedefs that return pointers"
This reverts commit 19ab6044be.
We will replace the commit with the fix from Linux.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-27-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:24 -05:00
Paolo Bonzini
47da500a7e Revert "kernel-doc: Handle function typedefs without asterisks"
This reverts commit 3cd3c5193c.
We will replace the commit with the fix from Linux.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-26-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:24 -05:00
Mauro Carvalho Chehab
ac3617d90c scripts: kernel-doc: try to use c:function if possible
There are a few namespace clashes by using c:macro everywhere:

basically, when using it, we can't have something like:

	.. c:struct:: pwm_capture

	.. c:macro:: pwm_capture

So, we need to use, instead:

	.. c:function:: int pwm_capture (struct pwm_device * pwm, struct pwm_capture * result, unsigned long timeout)

for the function declaration.

The kernel-doc change was proposed by Jakob Lykke Andersen here:

	6fd2076ec0

Although I did a different implementation.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-25-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:24 -05:00
Mauro Carvalho Chehab
4f5f16a193 scripts: kernel-doc: fix line number handling
Address several issues related to pointing to the wrong line
number:

1) ensure that line numbers will always be initialized

   When section is the default (Description), the line number
   is not initializing, producing this:

	$ ./scripts/kernel-doc --enable-lineno ./drivers/media/v4l2-core/v4l2-mem2mem.c|less

	**Description**

	#define LINENO 0
	In case of streamoff or release called on any context,
	1] If the context is currently running, then abort job will be called
	2] If the context is queued, then the context will be removed from
	   the job_queue

  Which is not right. Ensure that the line number will always
  be there. After applied, the result now points to the right location:

	**Description**

	#define LINENO 410
	In case of streamoff or release called on any context,
	1] If the context is currently running, then abort job will be called
	2] If the context is queued, then the context will be removed from
	   the job_queue

2) The line numbers for function prototypes are always + 1,
   because it is taken at the line after handling the prototype.
   Change the logic to point to the next line after the /** */
   block;

3) The "DOC:" line number should point to the same line as this
   markup is found, and not to the next one.

Probably part of the issues were due to a but that was causing
the line number offset to be incremented by one, if --export
were used.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-24-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:24 -05:00
Mauro Carvalho Chehab
486966e4a4 scripts: kernel-doc: allow passing desired Sphinx C domain dialect
When kernel-doc is called via kerneldoc.py, there's no need to
auto-detect the Sphinx version, as the Sphinx module already
knows it. So, add an optional parameter to allow changing the
Sphinx dialect.

As kernel-doc can also be manually called, keep the auto-detection
logic if the parameter was not specified. On such case, emit
a warning if sphinx-build can't be found at PATH.

I ended using a suggestion from Joe for using a more readable
regex, instead of using a complex one with a hidden group like:

	m/^(\d+)\.(\d+)(?:\.?(\d+)?)/

in order to get the optional <patch> argument.

Thanks-to: Joe Perches <joe@perches.com>
Suggested-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-23-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:23 -05:00
Mauro Carvalho Chehab
0c77185233 scripts: kernel-doc: don't mangle with parameter list
While kernel-doc needs to parse parameters in order to
identify its name, it shouldn't be touching the type,
as parsing it is very difficult, and errors happen.

One current error is when parsing this parameter:

	const u32 (*tab)[256]

Found at ./lib/crc32.c, on this function:

	u32 __pure crc32_be_generic (u32 crc, unsigned char const *p, size_t len, const u32 (*tab)[256], u32 polynomial);

The current logic mangles it, producing this output:

	const u32 ( *tab

That's something that it is not recognizeable.

So, instead, let's push the argument as-is, and use it
when printing the function prototype and when describing
each argument.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-22-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:23 -05:00
Mauro Carvalho Chehab
3999ffcf13 scripts: kernel-doc: fix typedef identification
Some typedef expressions are output as normal functions.

As we need to be clearer about the type with Sphinx 3.x,
detect such cases.

While here, fix a wrongly-indented block.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-21-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:23 -05:00
Mauro Carvalho Chehab
78c8c92c5d scripts: kernel-doc: reimplement -nofunction argument
Right now, the build system doesn't use -nofunction, as
it is pretty much useless, because it doesn't consider
the other output modes (extern, internal), working only
with all.

Also, it is limited to exclude functions.

Re-implement it in order to allow excluding any symbols from
the document output, no matter what mode is used.

The parameter was also renamed to "-nosymbol", as it express
better its meaning.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201117165312.118257-20-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-12-10 12:15:23 -05:00