So far, we expect from a commit that renames a file to contain a
changelog entry only for the new name. For example, after the following
commit:
$ git move foo bar
$ git commit
We expect the following changelog:
* bar: Renamed from foo.
Git does not keep track of renamings, only file deletions and additions.
The display of patches then uses heuristics (with config-dependent
parameters) to try to match deleted and added files in the same commit.
It is thus brittle to rely on this information.
This commit modifies changelog processing so that renames are considered
as a deletion of a file plus an addition of another file. The following
changelog is now expected for the above example:
* foo: Move...
* bar: Here.
contrib/
* gcc-changelog/git_email.py (GitEmail.__init__): Interpret file
renamings as a file deletion plus a file addition.
* gcc-changelog/git_repository.py (parse_git_revisions):
Likewise.
* gcc-changelog/test_email.py: New testcase.
* gcc-changelog/test_patches.txt: New testcase.
This extends the ChangeLog entries parsing machinery to handle entries
that cover multiple files spanning over multiple lines. For instance:
* first_file_patched.c, second_file_patched.c, third_file_patched.c,
fourth_file_patched.c: Do things.
contrib/
* gcc-changelog/git_commit.py (ChangeLogEntry): Handle entries
with multi-line file lists.
* gcc-changelog/test_email.py: New testcase.
* gcc-changelog/test_patches.txt: Likewise.
Currently, running gcc-changelog's unit tests may clutter the output
with tons of warnings such as:
.../contrib/gcc-changelog/git_email.py:40: ResourceWarning: unclosed
file <_io.TextIOWrapper name='/tmp/tmpt5okd4qp.patch' mode='r'
encoding='UTF-8'>
lines = open(self.filename).read().splitlines()
ResourceWarning: Enable tracemalloc to get the object allocation
traceback
This commit fixes these leaks, which restores a clean testsuite output.
contrib/
* gcc-changelog/git_update_version.py: Close file objects after
use.
* gcc-changelog/git_email.py: Likewise.
* gcc-changelog/test_email.py: Likewise.
The patch improves the script based on Jakub's needs,
I'm going to install the patch.
contrib/ChangeLog:
* gcc-changelog/git_update_version.py: Add 2 new options.
I've long introduced ChangeLog entries as "for dir/ChangeLog", a
format adopted by GNU CVS-Utilities some 20 years ago. My commits
have been formatted like this forever.
This patch makes it acceptable for git gcc-verify.
contrib/ChangeLog:
* gcc-changelog/git_commit.py (changelog_regex): Accept optional
'for' prefix.
This patch simplifies (!!!) the logic governing the naming of dump
files and auxiliary output files in the driver, in the compiler, and
in the LTO wrapper. No changes are made to the naming of primary
outputs, there are often ways to restore past behavior, and a number
of inconsistencies are fixed. Some internal options are removed
(-auxbase and -auxbase-strip), sensible existing uses of -dumpdir and
-dumpbase options remain unchanged, additional useful cases are added,
making for what is still admittedly quite complex. Extensive
documentation and testcases provide numerous examples, from normal to
corner cases.
The most visible changes are:
- aux and dump files now always go in the same directory, that
defaults to the directory of the primary output, but that can be
overridden with -dumpdir, -save-temps=*, or, preserving past behavior,
with a -dumpbase with a directory component.
- driver and compiler now have the same notion of naming of auxiliary
outputs, e.g. .dwo files will no longer be in one location while the
debug info suggests they are elsewhere, and -save-temps and .dwo
auxiliary outputs now go in the same location as .su, .ci and
coverage data, with consistent naming.
- explicitly-specified primary output names guide not only the
location of aux and dump outputs: the output base name is also used in
their base name, as a prefix when also linking (e.g. foo.c bar.c -o
foobar creates foobar-foo.dwo and foobar-bar.dwo with -gsplit-dwarf),
or as the base name instead of the input name (foo.c -c -o whatever.o
creates whatever.su rather than foo.su with -fstack-usage). The
preference for the input file base name, quite useful for our
testsuite, can be restored with -dumpbase "". When compiling and
linking tests in the testsuite with additional inputs, we now use this
flag. Files named in dejagnu board ldflags, libs, and ldscripts are
now quoted in the gcc testsuite with -Wl, so that they are not counted
as additional inputs by the compiler driver.
- naming a -dumpbase when compiling multiple sources used to cause
dumps from later compiles to overwrite those of earlier ones; it is
now used as a prefix when compiling multiple sources, like an
executable name above.
- the dumpbase, explicitly specified or computed from output or input
names, now also governs the naming of aux outputs; since aux outputs
usually replaced the suffix from the input name, while dump outputs
append their own additional suffixes, a -dumpbase-ext option is
introduced to enable a chosen suffix to be dropped from dumpbase to
form aux output names.
- LTO dump and aux outputs were quite a mess, sometimes leaking
temporary output names into -save-temps output names, sometimes
conversely generating desirable aux outputs in temporary locations.
They now obey the same logic of compiler aux and dump outputs, landing
in the expected location and taking the linker output name or an
explicit dumpbase overrider into account.
- Naming of -fdump-final-insns outputs now follows the dump file
naming logic for the .gkd files, and the .gk dump files generated in
the second -fcompare-debug compilation get the .gk inserted before the
suffix that -dumpbase-ext drops in aux outputs.
gcc/ChangeLog:
* common.opt (aux_base_name): Define.
(dumpbase, dumpdir): Mark as Driver options.
(-dumpbase, -dumpdir): Likewise.
(dumpbase-ext, -dumpbase-ext): New.
(auxbase, auxbase-strip): Drop.
* doc/invoke.texi (-dumpbase, -dumpbase-ext, -dumpdir):
Document.
(-o): Introduce the notion of primary output, mention it
influences auxiliary and dump output names as well, add
examples.
(-save-temps): Adjust, move examples into -dump*.
(-save-temps=cwd, -save-temps=obj): Likewise.
(-fdump-final-insns): Adjust.
* dwarf2out.c (gen_producer_string): Drop auxbase and
auxbase_strip; add dumpbase_ext.
* gcc.c (enum save_temps): Add SAVE_TEMPS_DUMP.
(save_temps_prefix, save_temps_length): Drop.
(save_temps_overrides_dumpdir): New.
(dumpdir, dumpbase, dumpbase_ext): New.
(dumpdir_length, dumpdir_trailing_dash_added): New.
(outbase, outbase_length): New.
(The Specs Language): Introduce %". Adjust %b and %B.
(ASM_FINAL_SPEC): Use %b.dwo for an aux output name always.
Precede object file with %w when it's the primary output.
(cpp_debug_options): Do not pass on incoming -dumpdir,
-dumpbase and -dumpbase-ext options; recompute them with
%:dumps.
(cc1_options): Drop auxbase with and without compare-debug;
use cpp_debug_options instead of dumpbase. Mark asm output
with %w when it's the primary output.
(static_spec_functions): Drop %:compare-debug-auxbase-opt and
%:replace-exception. Add %:dumps.
(driver_handle_option): Implement -save-temps=*/-dumpdir
mutual overriding logic. Save dumpdir, dumpbase and
dumpbase-ext options. Do not save output_file in
save_temps_prefix.
(adds_single_suffix_p): New.
(single_input_file_index): New.
(process_command): Combine output dir, output base name, and
dumpbase into dumpdir and outbase.
(set_collect_gcc_options): Pass a possibly-adjusted -dumpdir.
(do_spec_1): Optionally dumpdir instead of save_temps_prefix,
and outbase instead of input_basename in %b, %B and in
-save-temps aux files. Handle empty argument %".
(driver::maybe_run_linker): Adjust dumpdir and auxbase.
(compare_debug_dump_opt_spec_function): Adjust gkd dump file
naming. Spec-quote the computed -fdump-final-insns file name.
(debug_auxbase_opt): Drop.
(compare_debug_self_opt_spec_function): Drop auxbase-strip
computation.
(compare_debug_auxbase_opt_spec_function): Drop.
(not_actual_file_p): New.
(replace_extension_spec_func): Drop.
(dumps_spec_func): New.
(convert_white_space): Split-out parts into...
(quote_string, whitespace_to_convert_p): ... these. New.
(quote_spec_char_p, quote_spec, quote_spec_arg): New.
(driver::finalize): Release and reset new variables; drop
removed ones.
* lto-wrapper.c (HAVE_TARGET_EXECUTABLE_SUFFIX): Define if...
(TARGET_EXECUTABLE_SUFFIX): ... is defined; define this to the
empty string otherwise.
(DUMPBASE_SUFFIX): Drop leading period.
(debug_objcopy): Use concat.
(run_gcc): Recognize -save-temps=* as -save-temps too. Obey
-dumpdir. Pass on empty dumpdir and dumpbase with a directory
component. Simplify temp file names.
* opts.c (finish_options): Drop aux base name handling.
(common_handle_option): Drop auxbase-strip handling.
* toplev.c (print_switch_values): Drop auxbase, add
dumpbase-ext.
(process_options): Derive aux_base_name from dump_base_name
and dump_base_ext.
(lang_dependent_init): Compute dump_base_ext along with
dump_base_name. Disable stack usage and callgraph-info during
lto generation and compare-debug recompilation.
gcc/fortran/ChangeLog:
* options.c (gfc_get_option_string): Drop auxbase, add
dumpbase_ext.
gcc/ada/ChangeLog:
* gcc-interface/lang-specs.h: Drop auxbase and auxbase-strip.
Use %:dumps instead of -dumpbase. Add %w for implicit .s
primary output.
* switch.adb (Is_Internal_GCC_Switch): Recognize dumpdir and
dumpbase-ext. Drop auxbase and auxbase-strip.
lto-plugin/ChangeLog:
* lto-plugin.c (skip_in_suffix): New.
(exec_lto_wrapper): Use skip_in_suffix and concat to build
non-temporary output names.
(onload): Look for -dumpdir in COLLECT_GCC_OPTIONS, and
override link_output_name with it.
contrib/ChangeLog:
* compare-debug: Adjust for .gkd files named as dump files,
with the source suffix rather than the object suffix.
gcc/testsuite/ChangeLog:
* gcc.misc-tests/outputs.exp: New.
* gcc.misc-tests/outputs-0.c: New.
* gcc.misc-tests/outputs-1.c: New.
* gcc.misc-tests/outputs-2.c: New.
* lib/gcc-defs.exp (gcc_adjusted_linker_flags): New.
(gcc_adjust_linker_flags): New.
(dg-additional-files-options): Call it. Pass -dumpbase ""
when there are additional sources.
* lib/profopt.exp (profopt-execute): Pass the executable
suffix with -dumpbase-ext.
* lib/scandump.exp (dump-base): Mention -dumpbase "" use.
* lib/scanltranstree.exp: Adjust dump suffix expectation.
* lib/scanwpaipa.exp: Likewise.
We should be able to generate ChangeLog entries for
commits like b3d566f570.
I'm going to install the patch.
contrib/ChangeLog:
* gcc-changelog/git_commit.py: Parse changelog entries for
ignored locations.
* gcc-changelog/test_email.py: Add new test for it.
* gcc-changelog/test_patches.txt: Likewise.
This patch introduces a prepare-commit-msg hook that appends a ChangeLog
skeleton to a commit message when the GCC_FORCE_MKLOG environment variable
is set, and a 'git commit-mklog' command set that variable while running
'git commit'.
contrib/ChangeLog:
* prepare-commit-msg: New file.
* gcc-git-customization.sh: Install it. Add commit-mklog alias.
* mklog.py: Add new option -c which appends
to a ChangeLog file.
* gcc-changelog/git_commit.py: Add trailing '/'
for libdruntime. Allow empty changelog for
only ignored files.
* gcc-changelog/test_email.py: New test for go
patch in ignored location.
* gcc-changelog/test_patches.txt: Add test.
2020-05-22 Jakub Jelinek <jakub@redhat.com>
* gcc-changelog/git_commit.py: Add trailing / to
gcc/testsuite/go.test/test and replace gcc/go/frontend/
with gcc/go/gofrontend/ in ignored locations.
* gcc-changelog/git_commit.py: Add author_tuple
only if not present in author_lines.
* gcc-changelog/test_email.py: New test.
* gcc-changelog/test_patches.txt: Add new patch.
* gcc-changelog/git_commit.py: Refactor to make flake8 happy.
* gcc-changelog/test_email.py: Add new test.
* gcc-changelog/test_patches.txt: Add new patch.
git-check-commit.py does not know about "Reviewed-by",
"Reviewed-on", and "Signed-off-by" lines and hence it
expects those lines which follow the ChangeLog entries
to be indented by a tab.
This commit makes the script skip those lines. No further
processing is attempted because the review information
is not part of the ChangeLogs.
contrib/
2020-05-20 Frederik Harwath <frederik@codesourcery.com>
* gcc-changelog/git_commit.py: Skip over lines starting
with "Reviewed-by: ", "Reviewed-on: ", or "Signed-off-by: "
* gcc-changelog/git_commit.py: Support cherry pick
prefix.
* gcc-changelog/test_email.py: Test it.
* gcc-changelog/test_patches.txt: Add new patch.
* gcc-git-customization.sh: Add gcc-backport.
contrib/ChangeLog:
2020-05-15 Martin Liska <mliska@suse.cz>
* gcc-git-customization.sh: Add
alias.gcc-mklog new hook.
* mklog.py: New file.
* test_mklog.py: New file.
This commit is for the benefit of GDB, but as the binutils-gdb
repository shares the contrib/ directory with gcc, this commit must
first be applied to gcc then copied back to binutils-gdb.
This commit extends the two scripts contrib/dg-extract-results.{py,sh}
to handle some new, GDB specific test result types. These test
results types should never appear in GCC, or any other tool that
shares the contrib/ directly, so this change should be harmless.
In this patch series:
https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html
changes were made in GDB's use of Dejagnu so that two additional
conditions could be detected, these are:
1. Test names that contain either the build or source paths. Such
test names make it difficult to compare the results of two test runs
of GDB from two different directories, and
2. Duplicate test names. Duplicates make it difficult to track down
exactly which test has failed.
When running Dejagnu on GDB we can now (sometimes) see two additional
test result types matching the above conditions, these are '# of paths
in test names' and '# of duplicate test names'.
If the test is run in parallel mode (make -j...) then these extra test
results will appear in the individual test summary files, but are not
merged into the final summary file.
Additionally, within the summary file there are now two new types of
test summary line, these are 'PATH: ...' and 'DUPLICATE: ...', these
allow users to quickly search the test summary to track down where the
offending test names are. These lines are similarly not merged into
the unified gdb.sum file after a parallel test run.
This commit extends the dg-extract-results.* scripts to calculate the
totals for the two new result types, and to copy the new test summary
lines into the unified summary file.
contrib/ChangeLog:
* dg-extract-results.py: Handle GDB specific test types.
* dg-extract-results.sh: Likewise.
* gcc-changelog/git_check_commit.py: New file.
* gcc-changelog/git_commit.py: New file.
* gcc-changelog/git_email.py: New file.
* gcc-changelog/git_repository.py: New file.
* gcc-changelog/git_update_version.py: New file.
As Iain reported, my change broke the case when one has bison >= 3,
but make decides there is no reason to regenerate plural.c, unfortunately
that seems to be a scenario I haven't tested. The problem is that
the pregenerated plural.c has been generated with bison 1.35, but when
config.h says HAVE_BISON3, the code assumes it is the bison3 variant.
What used to work fine is when one has bison >= 3 and plural.c has been
regenerated (e.g. do touch intl/plural.y and it will work), or when
one doesn't have any bison (then nothing is regenerated, but HAVE_BISON3
isn't defined either), or when one has bison < 3 and doesn't need to
regenerate, or when one has bison < 3 and it is regenerated.
The following patch fixes this, by killing the HAVE_BISON3 macro from
config.h, and instead remembering the fact whether plural.c has been created
with bison < 3 or bison >= 3 in a separate new plural-config.h header.
The way this works:
- user doesn't have bison
- user has bison >= 3, but intl/{plural-config.h,plural.c} aren't older than intl/plural.y
- user has bison < 3, but intl/{plural-config.h,plural.c} aren't older than intl/plural.y
pregenerated !USE_BISON3 plural.c and plural-config.h from source
dir is used, nothing in the objdir
- user has bison >= 3 and intl/plural.y is newer
Makefile generates plural.c and USE_BISON3 plural-config.h in the
objdir, which is then used in preference to srcdir copies
- user has bison < 3 and intl/plural.y is newer
Makefile generates plural.c and !USE_BISON3 plural-config.h in the
objdir, which is then used in preference to srcdir copies
I have tested all these cases and make all-yes worked in all the cases.
If one uses the unsupported ./configure where srcdir == objdir, I guess
(though haven't tested) that it should still work, just it would be nice
if such people didn't try to check in the plural{.c,-config.h} they have
regenerated.
What doesn't work, but didn't work before either (just tested gcc-9 branch
too) is when one doesn't have bison and plural.y is newer than plural.c.
Don't do that ;)
2020-04-16 Jakub Jelinek <jakub@redhat.com>
PR bootstrap/92008
intl/
* configure.ac: Remove HAVE_BISON3 AC_DEFINE.
* Makefile.in (HEADERS): Add plural-config.h.
(.y.c): Also create plural-config.h.
(dcigettext.o loadmsgcat.o plural.o plural-exp.o): Also depend
on plural-config.h.
(plural-config.h): Depend on plural.c.
* plural-exp.h: Include plural-config.h. Use USE_BISON3 instead
of HAVE_BISON3.
* plural.y: Use USE_BISON3 instead of HAVE_BISON3.
* configure: Regenerated.
* plural.c: Regenerated.
* config.h.in: Regenerated.
* plural-config.h: Generated.
contrib/
* gcc_update: Add intl/plural.y dependency for intl/plural-config.h.
Whilst trying to convert the add vendor branch script to work with
personal branches I encountered a minor issue where git would report
ambiguous refs when checking out the new branch.
It turns out that this is because git considers <me>/<topic> to be
ambiguous if both
refs/heads/<me>/<topic>
and
refs/remotes/<me>/<topic>
exist in the list of known branches.
Having thought about this a bit, I think the best solution is to adopt
something more like the vendors space and call the remote users/<me>
(this also works better if you want to set up remotes to track other
users branches as well).
So this patch has two parts.
1) It updates gcc-git-customization.sh to set up the new 'remote' and
converts any existing remote and branches tracking that to the new
format
2) It adds a new script to set up a personal branch on the gcc git repository.
* gcc-git-customization.sh: Use users/<pfx> for the personal remote
rather than just <pfx>. Convert any existing personal branches to the
new remote.
* git-add-user-branch.sh: New file.
This script is intended to create a new vendor branch. Doing so is
not completely obvious if you are not familiar with the upstream
structure, so this takes the pain out of getting it right.
It doesn't check out the branch locally, but does set everything up so
that, if you have push enabled for your vendor branches, then
git push vendors/<vendor> <branch>
will work as expected.
Run the script as
contrib/git-add-vendor-branch.sh <vendor>/<branch> <start-point>
the <vendor> space must have previously been set up in the way
git-fetch-vendor.sh expects.
* git-add-vendor-branch.sh: New file.
The initial structure for vendor and personal branches makes use of
the default remote (normally origin) for the upstream
repository). Unfortunately, this causes some confusion, especially for
personal branches because a push will not push to the correct upstream
location. This can be 'fixed' by adding a push refspec for the remote,
but that has the unfortunate consequence of breaking the push.default
behaviour for git push, and it becomes too easy to accidentally commit
something unintended to the main parts of the repository.
To work around this, this patch changes the configuration to use
separate 'remotes' for these additional refs, with one remote for the
personal space and another remote for each vendor's space. The
personal space is called after the user's preferred branch-space
prefix (default 'me'), the vendor spaces are called
vendors/<vendor-name>.
As far as possible, I've made the script automatically restructure any
existing fetch or push lines that earlier versions of the scripts may
have created - the gcc-git-customization.sh script will convert all
vendor refs that it can find, so it is not necessary to re-add any
vendors you've already added.
You might, however, want to run
git remote prune <origin>
after running to clean up any stale upstream-refs that might still be
in your local repo, and then
git fetch vendors/<vendor>
or
git fetch <me>
to re-populate the remotes/ structures.
Also, for any branch you already have that tracks a personal or vendor
branch upstream, you might need to run
git config branch.<name>.remote <new-remote>
so that merges and pushes go to the right place (I haven't attempted
to automate this last part).
For vendors, the new structure means that
git checkout -b <vendor>/<branch> remotes/vendors/<vendor>/<branch>
will correctly set up a remote tracking branch.
Please be aware that if you have multiple personal branches set up, then
git push <me>
will still consider all of them for pushing. If you only want to push
one branch, then either write
git push <me> HEAD
or
git push <me> <me>/branch
as appropriate.
And don't forget '-n' (--dry-run) to see what would be done if this
were not a dry run.
Finally, now that the vendors spaces are isolated from each other and
from the other spaces, I've added an option "--enable-push" to
git-fetch-vendor.sh. If passed, then a "push" spec will be added for
that vendor to enable pushing to the upstream. If you re-run the
script for the same vendor without the option, the push spec will be
removed.
* gcc-git-customization.sh: Check that user-supplied remote
name exists before continuting. Use a separate remotes for the
personal commit area. Convert existing personal and vendor
fetch rules to new layout.
* git-fetch-vendor.sh: New vendor layout. Add --enable-push
option.
Really old git versions (like 1.6.0) require
"git log --pretty=tformat:%p:%t:%H"
or else we see:
Updating GIT tree
Current branch master is up to date.
fatal: invalid --pretty format: %p:%t:%H
Adjusting file timestamps
Touching gcc/config.in...
Touching gcc/config/arm/arm-tune.md...
...and an empty revision in LAST_UPDATED and gcc/REVISION.
In its absence, for newer git versions, "tformat" is the default
qualifier, documented as such default for at least git-2.11.0.
