preprocessor: Fix cpp_avoid_paste for digit separators

The libcpp function cpp_avoid_paste is used to insert whitespace in
preprocessed output where needed to avoid two consecutive
preprocessing tokens, that logically (e.g. when stringized) do not
have whitespace between them, from being incorrectly lexed as one when
the preprocessed input is reread by a compiler.

This fails to allow for digit separators, so meaning that invalid
code, that has a CPP_NUMBER (from a macro expansion) followed by a
character literal, can result in preprocessed output with a valid use
of digit separators, so that required syntax errors do not occur when
compiling with -save-temps.  Fix this by handling that case in
cpp_avoid_paste (as with other cases in cpp_avoid_paste, this doesn't
try to check whether the language version in use supports digit
separators; it's always OK to have unnecessary whitespace in
preprocessed output).

Note: there are other cases, with various kinds of wide character or
string literal following a CPP_NUMBER, where spurious pasting of
preprocessing tokens can occur but the sequence of tokens remains
invalid both before and after that pasting.  Maybe cpp_avoid_paste
should also handle those cases (and similar cases after a CPP_NAME),
to ensure the sequence of preprocessing tokens in preprocessed output
is exactly right, whether or not it affects whether syntax errors
occur.  This patch only addresses the case with digit separators where
invalid code can fail to be diagnosed without the space inserted.

Bootstrapped with no regressions for x86_64-pc-linux-gnu.

libcpp/
	* lex.c (cpp_avoid_paste): Do not allow pasting CPP_NUMBER with
	CPP_CHAR.

gcc/testsuite/
	* g++.dg/cpp1y/digit-sep-paste.C, gcc.dg/c2x-digit-separators-3.c:
	New tests.
This commit is contained in:
Joseph Myers 2021-05-11 18:54:32 +00:00
parent 21dfb22920
commit 3e3fdf3d52
3 changed files with 24 additions and 0 deletions

View File

@ -0,0 +1,11 @@
// Test token pasting with digit separators avoided for preprocessed output.
// { dg-do compile { target c++14 } }
// { dg-options "-save-temps" }
#define ZERO 0
int
f ()
{
return ZERO'0'0; /* { dg-error "expected" } */
}

View File

@ -0,0 +1,12 @@
/* Test C2x digit separators. Test token pasting avoided for preprocessed
output. */
/* { dg-do compile } */
/* { dg-options "-std=c2x -save-temps" } */
#define ZERO 0
int
f (void)
{
return ZERO'0'0; /* { dg-error "expected" } */
}

View File

@ -3725,6 +3725,7 @@ cpp_avoid_paste (cpp_reader *pfile, const cpp_token *token1,
|| b == CPP_NAME
|| b == CPP_CHAR || b == CPP_STRING); /* L */
case CPP_NUMBER: return (b == CPP_NUMBER || b == CPP_NAME
|| b == CPP_CHAR
|| c == '.' || c == '+' || c == '-');
/* UCNs */
case CPP_OTHER: return ((token1->val.str.text[0] == '\\'