appsrc->avimux->appsink - very slow

[Nicholas Butts]

I'm capturing 720P video 20 FPS in a MPEG-4-ES stream, so I have 50ms to
capture and process a frame. I think my problem has to do with the time
stamps. I can write the raw ES streams to disk and then I can play it back
without any problems. But when I read the raw frames and try to mux it, then
get the data back our and write it I get corrupted frames.

Here is the code I am using to push buffers into the pipeline:
    GstBuffer * vidbuf = gst_buffer_try_new_and_alloc(vidLength);
    if (vidbuf == NULL)
    {
        Log::logError("[VideoMuxer::mux] Failed to allocate a video
buffer");
    }
    GST_BUFFER_DURATION(vidbuf) = GST_CLOCK_TIME_NONE;
    GST_BUFFER_TIMESTAMP(vidbuf) = GST_CLOCK_TIME_NONE;
    GST_BUFFER_OFFSET(vidbuf) = GST_BUFFER_OFFSET_NONE;
    GST_BUFFER_OFFSET_END(vidbuf) = GST_BUFFER_OFFSET_NONE;
//    GST_BUFFER_DURATION(vidbuf) = nsDuration;
//    GST_BUFFER_TIMESTAMP(vidbuf) = nsTime;
    gst_buffer_set_data(vidbuf, vid, vidLength);
    error = gst_app_src_push_buffer(GST_APP_SRC(_videoSrc), vidbuf);
    if (error != GST_FLOW_OK)
    {
        Log::logError("[VideoMuxer::mux] Failed to push data into
_videoSrc");
    }

I've commented out the code that had the duration and current time. It would
generate garbage data if I kept the time stamps. The way the code is written
above it will not produce any data into a file. I can get this to work on
the PC by reading the MPEG4-ES data and parsing frames from this file and
then pushing it into the same pipeline. But when I do this on the embedded
system it doesn't work.

Another odd thing is the MPEG-4 hardware compressor produces weird frame
times. Here is what ffprobe says about my MPEG4-ES streams. Note that this
example uses 24 FPS instead of the 20 FPS of the production system:
Input #0, m4v, from '2010_12_17_Good.m4v':
  Duration: N/A, bitrate: N/A
    Stream #0.0: Video: mpeg4, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 24k
tbr, 1200k tbn, 24k tbc
[STREAM]
codec_name=mpeg4
codec_long_name=MPEG-4 part 2
decoder_time_base=1/24000
codec_type=video
r_frame_rate=24000.000000
r_frame_rate_num=24000
r_frame_rate_den=1
width=1280
height=720
gop_size=12
has_b_frames=1
sample_aspect_ratio=1/1
display_aspect_ratio=16/9
pix_fmt=yuv420p
index=0
time_base=1/1200000
start_time=N/A
duration=N/A
nb_frames=0
[/STREAM]

Now when I mux this into a MOV file (based on an earlier post about the
inadequacies of AVI files). I get the following from ffprobe:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mov':
  Duration: 00:00:03.69, start: 0.000000, bitrate: 3517 kb/s
    Stream #0.0(eng): Video: mpeg4, yuv420p, 1280x720 [PAR 1:1 DAR 16:9],
23.83 tbr, 2400 tbn, 24k tbc
    Stream #0.1(eng): Audio: pcm_s16le, 32000 Hz, stereo, s16, 1024 kb/s
[STREAM]
codec_name=mpeg4
codec_long_name=MPEG-4 part 2
decoder_time_base=1/24000
codec_type=video
language=p�F    ��F
r_frame_rate=23.833333
r_frame_rate_num=143
r_frame_rate_den=6
width=1280
height=720
gop_size=12
has_b_frames=1
sample_aspect_ratio=1/1
display_aspect_ratio=16/9
pix_fmt=yuv420p
index=0
time_base=1/2400
language=eng
start_time=0.000000
duration=3.654167
nb_frames=0
[/STREAM]
[STREAM]
codec_name=pcm_s16le
codec_long_name=PCM signed 16-bit little-endian
decoder_time_base=0/1
codec_type=audio
language=��F    p�F
sample_rate=32000.000000
channels=2
bits_per_sample=16
index=1
time_base=1/32000
language=eng
start_time=0.000000
duration=3.696000
nb_frames=0
[/STREAM]


I suspect I am doing something wrong with the time stamps of the data. Can
anyone help with an example of how to correctly mux an already compressed
video and audio streams together?

Thanks in advance!!!



On Thu, Aug 25, 2011 at 6:39 PM, Stefan Kost <ensonic at hora-obscura.de>wrote:

> On 08/25/11 23:28, Nicholas Butts wrote:
> > I'm trying to mux an MPEG4-ES stream with an uncompressed audio stream
> > into an avimux filter and then to an appsink. I have this working on
> > Ubuntu 10.04. Now I am moving to an ARM based embedded system and this
> > is REALLY slow. It takes on-average 12.7 ms to performing the muxing.
> > This seems extremely slow to me. Has anyone else attempted this? Is
> > there a way to disable all of the threading and make a single threaded
> > application?
> >
> Even if you could disable the threading, I doubt that this would be the
> source of the slowness. Check that you don't copy data where it can be
> avoided. Also using appsrc and appsink would make me think that your
> design isn't that good.
>
> Stefan
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20110826/60abc551/attachment.htm>