OpenE2K/binutils-gdb
12
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Prevent overflow in rl_set_screen_size

Browse Source 
GDB calls rl_set_screen_size in readline with the current screen size,
measured in rows and columns.  To represent "infinite" sizes, GDB
passes in INT_MAX; however, since rl_set_screen_size internally
multiplies the number of rows and columns, this causes a signed
integer overflow.  To prevent this we can instead pass in the
approximate square root of INT_MAX (which is still reasonably large),
so that even when the number of rows and columns is "infinite" we
don't overflow.

gdb/ChangeLog:
2019-02-27  Saagar Jha  <saagar@saagarjha.com>
	    Pedro Alves  <palves@redhat.com>

	* utils.c (set_screen_size): Reduce "infinite" rows and columns
	before calling rl_set_screen_size.
This commit is contained in:
Saagar Jha 2019-02-27 18:48:35 +00:00 committed by Pedro Alves
parent 6c28e44a35
commit 23031e3192
2 changed files with 23 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
gdb/ChangeLog
View File

@ -1,3 +1,9 @@
2019-02-27 Saagar Jha <saagar@saagarjha.com>
Pedro Alves <palves@redhat.com>
* utils.c (set_screen_size): Reduce "infinite" rows and columns
before calling rl_set_screen_size.
2019-02-27 Tom Tromey <tromey@adacore.com>
* configure.ac (HAVE_LIBPYTHON2_4, HAVE_LIBPYTHON2_5): Never

21
gdb/utils.c
View File

@ -1380,11 +1380,24 @@ set_screen_size (void)
int rows = lines_per_page;
int cols = chars_per_line;
if (rows <= 0)
rows = INT_MAX;
/* If we get 0 or negative ROWS or COLS, treat as "infinite" size.
A negative number can be seen here with the "set width/height"
commands and either:
if (cols <= 0)
cols = INT_MAX;
- the user specified "unlimited", which maps to UINT_MAX, or
- the user spedified some number between INT_MAX and UINT_MAX.
Cap "infinity" to approximately sqrt(INT_MAX) so that we don't
overflow in rl_set_screen_size, which multiplies rows and columns
to compute the number of characters on the screen. */
const int sqrt_int_max = INT_MAX >> (sizeof (int) * 8 / 2);
if (rows <= 0 || rows > sqrt_int_max)
rows = sqrt_int_max;
if (cols <= 0 || cols > sqrt_int_max)
cols = sqrt_int_max;
/* Update Readline's idea of the terminal size. */
rl_set_screen_size (rows, cols);