Attempting to debug rfbsrc plugin

[Peter Watkins]

gstreamer-devel-request at lists.freedesktop.org 
Sent by: 
gstreamer-devel-bounces+peter.watkins=biotronik.com at lists.freedesktop.org
08/03/2011 12:57 PM
Please respond to
gstreamer-devel at lists.freedesktop.org


To
gstreamer-devel at lists.freedesktop.org
cc

Subject
Attempting to debug rfbsrc plugin






I'd like to use the rfbsrc plugin for a project I'm working on. I know 
it's a "bad" plugin, but it does exactly what I want. To test it, I ran 
gst-launch with the following command:

OK. I got my backtrace. It turns out that I can't connect to the process. 
I have to run gst-launch from within GDB directly. Odd.

Anyway, here's the first part of the backtrace:
GLib-ERROR **: /build/buildd/glib2.0-2.28.6/./glib/gmem.c:170: failed to 
allocate 3042077755 bytes
aborting...

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb72d5b70 (LWP 2587)]
0x0012e416 in ?? ()
(gdb) bt
#0  0x0012e416 in ?? ()
#1  0x0035de71 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0x0036134e in abort () at abort.c:92
#3  0x0028cf27 in g_logv () from /lib/i386-linux-gnu/libglib-2.0.so.0
#4  0x0028cf62 in g_log () from /lib/i386-linux-gnu/libglib-2.0.so.0
#5  0x0028ab3d in g_malloc () from /lib/i386-linux-gnu/libglib-2.0.so.0
#6  0x00680ef0 in rfb_decoder_read (decoder=0x8058d38, 
    len=<value optimized out>) at rfbdecoder.c:174
#7  0x006810db in rfb_decoder_state_server_cut_text (decoder=0x8058d38)
    at rfbdecoder.c:900

That makes sense, GLib can't allocate ~ 3GB of memory on my little 
machine. Here's the offending code:

(gdb) f 6
#6  0x00680ef0 in rfb_decoder_read (decoder=0x8058d38, 
    len=<value optimized out>) at rfbdecoder.c:174
174         decoder->data = g_malloc (len);
(gdb) info args
decoder = 0x8058d38
len = <value optimized out>
(gdb) list
169       g_return_val_if_fail (len > 0, NULL);
170 
171       if (G_UNLIKELY (len > decoder->data_len)) {
172         if (decoder->data)
173           g_free (decoder->data);
174         decoder->data = g_malloc (len);
175         decoder->data_len = len;
176       }
177 
178       while (total < len) {


Too bad len is optimized out. Perhaps it's in a register somewhere. In any 
case, here's the caller to  rfb_decoder_read():

(gdb) f 7
#7  0x006810db in rfb_decoder_state_server_cut_text (decoder=0x8058d38)
    at rfbdecoder.c:900
900       rfb_decoder_read (decoder, cut_text_length);
(gdb) list
895 
896       /* 3 bytes padding, 4 bytes cut_text_length */
897       rfb_decoder_read (decoder, 7);
898       cut_text_length = RFB_GET_UINT32 (decoder->data + 3);
899 
900       rfb_decoder_read (decoder, cut_text_length);
901       GST_DEBUG ("rfb_decoder_state_server_cut_text: throw away '%s'",
902           decoder->data);
903 
904       decoder->state = rfb_decoder_state_normal;
(gdb) info locals
cut_text_length = <value optimized out>
__FUNCTION__ = "rfb_decoder_state_server_cut_text"

cut_text_length is optimized out too, but it seems like I should be able 
to determine what it is:

(gdb) print decoder->data
$12 = (guint8 *) 0x80b9900 "(\026I"