No subject


Wed Aug 3 17:12:22 PDT 2011 

(gdb) print ntohl(* (uint32_t *) (decoder->data + 3))
$10 = 2627145

However, that yelds 2627145, not 3042077755 (0xB5526C3B).

If I look in the registers for frame 5 (g_malloc ()), I can find 
0xB5526C3B in the Source Index register (ESI):

(gdb) f 5
#5  0x0028ab3d in g_malloc () from /lib/i386-linux-gnu/libglib-2.0.so.0
(gdb) info registers
eax            0x0      0
ecx            0xa1b    2587
edx            0x6      6
ebx            0x318ff4 3248116
esp            0xb72d4f40       0xb72d4f40
ebp            0xb72d4f68       0xb72d4f68
esi            0xb5526c3b       -1252889541
edi            0x8058d38        134581560
eip            0x28ab3d 0x28ab3d <g_malloc+109>
eflags         0x200206 [ PF IF ID ]
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51

ESI has a different value in frame 6:

(gdb) f 6 
#6  0x00680ef0 in rfb_decoder_read (decoder=0x8058d38, 
    len=<value optimized out>) at rfbdecoder.c:174
(gdb) info registers
eax            0x0      0
ecx            0xa1b    2587
edx            0x6      6
ebx            0x686ff4 6844404
esp            0xb72d4f70       0xb72d4f70
ebp            0xb72d4fb8       0xb72d4fb8
esi            0x8058d38        134581560
edi            0x8058d38        134581560
eip            0x680ef0 0x680ef0 <rfb_decoder_read+288>
eflags         0x200206 [ PF IF ID ]
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51

Alas, I'm not good enough at reading assembly to figure out where ESI gets 
changed between frame 6 and 5.

--=_alternative 0002BD51882578E2_=
Content-Type: text/html; charset="US-ASCII"


<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>gstreamer-devel-request at lists.freedesktop.org</b>
</font>
<br><font size=1 face="sans-serif">Sent by: gstreamer-devel-bounces+peter.watkins=biotronik.com at lists.freedesktop.org</font>
<p><font size=1 face="sans-serif">08/03/2011 12:57 PM</font>
<table border>
<tr valign=top>
<td bgcolor=white>
<div align=center><font size=1 face="sans-serif">Please respond to<br>
gstreamer-devel at lists.freedesktop.org</font></div></table>
<br>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">gstreamer-devel at lists.freedesktop.org</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Attempting to debug rfbsrc plugin</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br><tt><font size=2><br>
<br>
I'd like to use the rfbsrc plugin for a project I'm working on. I know
<br>
it's a &quot;bad&quot; plugin, but it does exactly what I want. To test
it, I ran <br>
gst-launch with the following command:</font></tt>
<br>
<br><font size=2 face="sans-serif">OK. I got my backtrace. It turns out
that I can't connect to the process. I have to run gst-launch from within
GDB directly. Odd.</font>
<br>
<br><font size=2 face="sans-serif">Anyway, here's the first part of the
backtrace:</font>
<br><font size=2 face="sans-serif">GLib-ERROR **: /build/buildd/glib2.0-2.28.6/./glib/gmem.c:170:
failed to allocate 3042077755 bytes</font>
<br><font size=2 face="sans-serif">aborting...</font>
<br>
<br><font size=2 face="sans-serif">Program received signal SIGABRT, Aborted.</font>
<br><font size=2 face="sans-serif">[Switching to Thread 0xb72d5b70 (LWP
2587)]</font>
<br><font size=2 face="sans-serif">0x0012e416 in ?? ()</font>
<br><font size=2 face="sans-serif">(gdb) bt</font>
<br><font size=2 face="sans-serif">#0 &nbsp;0x0012e416 in ?? ()</font>
<br><font size=2 face="sans-serif">#1 &nbsp;0x0035de71 in raise (sig=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:64</font>
<br><font size=2 face="sans-serif">#2 &nbsp;0x0036134e in abort () at abort.c:92</font>
<br><font size=2 face="sans-serif">#3 &nbsp;0x0028cf27 in g_logv () from
/lib/i386-linux-gnu/libglib-2.0.so.0</font>
<br><font size=2 face="sans-serif">#4 &nbsp;0x0028cf62 in g_log () from
/lib/i386-linux-gnu/libglib-2.0.so.0</font>
<br><font size=2 face="sans-serif">#5 &nbsp;0x0028ab3d in g_malloc () from
/lib/i386-linux-gnu/libglib-2.0.so.0</font>
<br><font size=2 face="sans-serif">#6 &nbsp;0x00680ef0 in rfb_decoder_read
(decoder=0x8058d38, </font>
<br><font size=2 face="sans-serif">&nbsp; &nbsp; len=&lt;value optimized
out&gt;) at rfbdecoder.c:174</font>
<br><font size=2 face="sans-serif">#7 &nbsp;0x006810db in rfb_decoder_state_server_cut_text
(decoder=0x8058d38)</font>
<br><font size=2 face="sans-serif">&nbsp; &nbsp; at rfbdecoder.c:900</font>
<br>
<br><font size=2 face="sans-serif">That makes sense, GLib can't allocate
~ 3GB of memory on my little machine. Here's the offending code:</font>
<br>
<br><font size=2 face="sans-serif">(gdb) f 6</font>
<br><font size=2 face="sans-serif">#6 &nbsp;0x00680ef0 in rfb_decoder_read
(decoder=0x8058d38, </font>
<br><font size=2 face="sans-serif">&nbsp; &nbsp; len=&lt;value optimized
out&gt;) at rfbdecoder.c:174</font>
<br><font size=2 face="sans-serif">174 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;decoder-&gt;data = g_malloc (len);</font>
<br><font size=2 face="sans-serif">(gdb) info args</font>
<br><font size=2 face="sans-serif">decoder = 0x8058d38</font>
<br><font size=2 face="sans-serif">len = &lt;value optimized out&gt;</font>
<br><font size=2 face="sans-serif">(gdb) list</font>
<br><font size=2 face="sans-serif">169 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;g_return_val_if_fail (len &gt; 0, NULL);</font>
<br><font size=2 face="sans-serif">170 &nbsp; &nbsp; &nbsp; &nbsp;</font>
<br><font size=2 face="sans-serif">171 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;if (G_UNLIKELY (len &gt; decoder-&gt;data_len)) {</font>
<br><font size=2 face="sans-serif">172 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;if (decoder-&gt;data)</font>
<br><font size=2 face="sans-serif">173 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;g_free (decoder-&gt;data);</font>
<br><font size=2 face="sans-serif">174 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;decoder-&gt;data = g_malloc (len);</font>
<br><font size=2 face="sans-serif">175 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;decoder-&gt;data_len = len;</font>
<br><font size=2 face="sans-serif">176 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;}</font>
<br><font size=2 face="sans-serif">177 &nbsp; &nbsp; &nbsp; &nbsp;</font>
<br><font size=2 face="sans-serif">178 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;while (total &lt; len) {</font>
<br>
<br>
<br><font size=2 face="sans-serif">Too bad len is optimized out. Perhaps
it's in a register somewhere. In any case, here's the caller to &nbsp;rfb_decoder_read():</font>
<br>
<br><font size=2 face="sans-serif">(gdb) f 7</font>
<br><font size=2 face="sans-serif">#7 &nbsp;0x006810db in rfb_decoder_state_server_cut_text
(decoder=0x8058d38)</font>
<br><font size=2 face="sans-serif">&nbsp; &nbsp; at rfbdecoder.c:900</font>
<br><font size=2 face="sans-serif">900 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;rfb_decoder_read (decoder, cut_text_length);</font>
<br><font size=2 face="sans-serif">(gdb) list</font>
<br><font size=2 face="sans-serif">895 &nbsp; &nbsp; &nbsp; &nbsp;</font>
<br><font size=2 face="sans-serif">896 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;/* 3 bytes padding, 4 bytes cut_text_length */</font>
<br><font size=2 face="sans-serif">897 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;rfb_decoder_read (decoder, 7);</font>
<br><font size=2 face="sans-serif">898 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;cut_text_length = RFB_GET_UINT32 (decoder-&gt;data + 3);</font>
<br><font size=2 face="sans-serif">899 &nbsp; &nbsp; &nbsp; &nbsp;</font>
<br><font size=2 face="sans-serif">900 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;rfb_decoder_read (decoder, cut_text_length);</font>
<br><font size=2 face="sans-serif">901 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;GST_DEBUG (&quot;rfb_decoder_state_server_cut_text: throw away '%s'&quot;,</font>
<br><font size=2 face="sans-serif">902 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;decoder-&gt;data);</font>
<br><font size=2 face="sans-serif">903 &nbsp; &nbsp; &nbsp; &nbsp;</font>
<br><font size=2 face="sans-serif">904 &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;decoder-&gt;state = rfb_decoder_state_normal;</font>
<br><font size=2 face="sans-serif">(gdb) info locals</font>
<br><font size=2 face="sans-serif">cut_text_length = &lt;value optimized
out&gt;</font>
<br><font size=2 face="sans-serif">__FUNCTION__ = &quot;rfb_decoder_state_server_cut_text&quot;</font>
<br>
<br><font size=2 face="sans-serif">cut_text_length is optimized out too,
but it seems like I should be able to determine what it is:</font>
<br>
<br><font size=2 face="sans-serif">(gdb) print decoder-&gt;data</font>
<br><font size=2 face="sans-serif">$12 = (guint8 *) 0x80b9900 &quot;(\026I&quot;</font>
<br>
<br><font size=2 face="sans-serif">From the description of RFB_GET_UINT32(),
it seems like it should do something like this:</font>
<br><font size=2 face="sans-serif">(gdb) print ntohl(* (uint32_t *) (decoder-&gt;data
+ 3))</font>
<br><font size=2 face="sans-serif">$10 = 2627145</font>
<br>
<br><font size=2 face="sans-serif">However, that yelds 2627145, not 3042077755
(0xB5526C3B).</font>
<br>
<br><font size=2 face="sans-serif">If I look in the registers for frame
5 (g_malloc ()), I can find 0xB5526C3B in the Source Index register (ESI):</font>
<br>
<br><font size=2 face="sans-serif">(gdb) f 5</font>
<br><font size=2 face="sans-serif">#5 &nbsp;0x0028ab3d in g_malloc () from
/lib/i386-linux-gnu/libglib-2.0.so.0</font>
<br><font size=2 face="sans-serif">(gdb) info registers</font>
<br><font size=2 face="sans-serif">eax &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x0 &nbsp; &nbsp; &nbsp; &nbsp;0</font>
<br><font size=2 face="sans-serif">ecx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xa1b &nbsp; &nbsp; &nbsp; &nbsp;2587</font>
<br><font size=2 face="sans-serif">edx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x6 &nbsp; &nbsp; &nbsp; &nbsp;6</font>
<br><font size=2 face="sans-serif">ebx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x318ff4 &nbsp; &nbsp; &nbsp; &nbsp;3248116</font>
<br><font size=2 face="sans-serif">esp &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xb72d4f40 &nbsp; &nbsp; &nbsp; &nbsp;0xb72d4f40</font>
<br><font size=2 face="sans-serif">ebp &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xb72d4f68 &nbsp; &nbsp; &nbsp; &nbsp;0xb72d4f68</font>
<br><font size=2 face="sans-serif">esi &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xb5526c3b &nbsp; &nbsp; &nbsp; &nbsp;-1252889541</font>
<br><font size=2 face="sans-serif">edi &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x8058d38 &nbsp; &nbsp; &nbsp; &nbsp;134581560</font>
<br><font size=2 face="sans-serif">eip &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x28ab3d &nbsp; &nbsp; &nbsp; &nbsp;0x28ab3d &lt;g_malloc+109&gt;</font>
<br><font size=2 face="sans-serif">eflags &nbsp; &nbsp; &nbsp; &nbsp; 0x200206
&nbsp; &nbsp; &nbsp; &nbsp;[ PF IF ID ]</font>
<br><font size=2 face="sans-serif">cs &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x73 &nbsp; &nbsp; &nbsp; &nbsp;115</font>
<br><font size=2 face="sans-serif">ss &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x7b &nbsp; &nbsp; &nbsp; &nbsp;123</font>
<br><font size=2 face="sans-serif">ds &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x7b &nbsp; &nbsp; &nbsp; &nbsp;123</font>
<br><font size=2 face="sans-serif">es &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x7b &nbsp; &nbsp; &nbsp; &nbsp;123</font>
<br><font size=2 face="sans-serif">fs &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x0 &nbsp; &nbsp; &nbsp; &nbsp;0</font>
<br><font size=2 face="sans-serif">gs &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x33 &nbsp; &nbsp; &nbsp; &nbsp;51</font>
<br>
<br><font size=2 face="sans-serif">ESI has a different value in frame 6:</font>
<br>
<br><font size=2 face="sans-serif">(gdb) f 6 &nbsp;</font>
<br><font size=2 face="sans-serif">#6 &nbsp;0x00680ef0 in rfb_decoder_read
(decoder=0x8058d38, </font>
<br><font size=2 face="sans-serif">&nbsp; &nbsp; len=&lt;value optimized
out&gt;) at rfbdecoder.c:174</font>
<br><font size=2 face="sans-serif">(gdb) info registers</font>
<br><font size=2 face="sans-serif">eax &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x0 &nbsp; &nbsp; &nbsp;0</font>
<br><font size=2 face="sans-serif">ecx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xa1b &nbsp; &nbsp;2587</font>
<br><font size=2 face="sans-serif">edx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x6 &nbsp; &nbsp; &nbsp;6</font>
<br><font size=2 face="sans-serif">ebx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x686ff4 6844404</font>
<br><font size=2 face="sans-serif">esp &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xb72d4f70 &nbsp; &nbsp; &nbsp; 0xb72d4f70</font>
<br><font size=2 face="sans-serif">ebp &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0xb72d4fb8 &nbsp; &nbsp; &nbsp; 0xb72d4fb8</font>
<br><font size=2 face="sans-serif">esi &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x8058d38 &nbsp; &nbsp; &nbsp; &nbsp;134581560</font>
<br><font size=2 face="sans-serif">edi &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x8058d38 &nbsp; &nbsp; &nbsp; &nbsp;134581560</font>
<br><font size=2 face="sans-serif">eip &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;0x680ef0 0x680ef0 &lt;rfb_decoder_read+288&gt;</font>
<br><font size=2 face="sans-serif">eflags &nbsp; &nbsp; &nbsp; &nbsp; 0x200206
[ PF IF ID ]</font>
<br><font size=2 face="sans-serif">cs &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x73 &nbsp; &nbsp; 115</font>
<br><font size=2 face="sans-serif">ss &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x7b &nbsp; &nbsp; 123</font>
<br><font size=2 face="sans-serif">ds &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x7b &nbsp; &nbsp; 123</font>
<br><font size=2 face="sans-serif">es &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x7b &nbsp; &nbsp; 123</font>
<br><font size=2 face="sans-serif">fs &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x0 &nbsp; &nbsp; &nbsp;0</font>
<br><font size=2 face="sans-serif">gs &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; 0x33 &nbsp; &nbsp; 51</font>
<br>
<br><font size=2 face="sans-serif">Alas, I'm not good enough at reading
assembly to figure out where ESI gets changed between frame 6 and 5.</font>
<br>
--=_alternative 0002BD51882578E2_=--

More information about the gstreamer-devel mailing list