Auto merge of #33612 - royalstream:royalstream-enc-enum-ptr, r=michaelwoerister

gdb Pretty Print: generic encoded was failing on reference/pointer types

If you debug this program using **gdb**
```rust
fn main() {
    let x = 10;
    let y = Some(&x);
    // additional code
}
```
And you try to print **y**'s value from the debugger, you get the following:
```
(gdb) print y
Python Exception <class 'gdb.error'> Cannot convert value to int.:
$1 = {RUST$ENCODED$ENUM$0$None = Some = {0x7fff5fbff97c}}
```
What happens is that inside **debugger_pretty_printers_common.py** the method `is_null_variant` doesn't have any special handling for pointer values so it ends up calling `.as_integer()` on `discriminant_val` (which holds a pointer) and fails.

Considering it needs to handle pointers and return _true_ when the pointer is _null_, I modified the `.as_integer()` method in **gdb_rust_pretty_printing.py** to take pointers into consideration.

After this modification **gdb** prints **y** like this:
```
(gdb) print y
$1 = Some = {0x7fff5fbff97c}
```
Now, it would be nice to print something useful (instead of a pointer address) but the pretty printer doesn't currently handle references/pointers so that's a completely different subject.
This commit is contained in:
bors 2016-05-15 05:26:50 -07:00
commit b358319353

View File

@ -70,6 +70,8 @@ class GdbValue(rustpp.Value):
return child
def as_integer(self):
if self.gdb_val.type.code == gdb.TYPE_CODE_PTR:
return int(str(self.gdb_val), 0)
return int(self.gdb_val)
def get_wrapped_value(self):