Multithreaded fast-forward possible?
Peter Rennert
p.rennert at cs.ucl.ac.uk
Thu Dec 13 13:55:40 PST 2012
I rewrote my script in C++ to make sure that I am not messing up things
with python, but it did not change anything. But I solved the problem.
If I compile the playback speed tutorial [1] that uses a playbin2 on my
file, I can do some kind of fast forward. If I test it with a file [2]
with low resolution, I can do fast-forward till about 90x (using
GST_SEEK_FLAG_SKIP does not change anything in my case). As I increase
the playback speed, the CPU use changes only slightly. But then suddenly
it seems that the threading overhead in ffdec_h264 becomes too big and
and a single thread goes to full load and blocks the playback. Fast
forward at about 50x starts to become instable and gets blocked some
time. I have the impression that flushing the pipeline helps in that
case. I will try tomorrow to install a handler that flushes the pipeline
in such cases.
Using my HD file, in the same setup allows me to speed the playback up
only until 4x speed. Even though the cores are under relative low use
(below 50%) one of the threads starts to block the playback. Anyway it
works somehow.
One remark to setting xvimagesink to sync=false. In some cases it makes
the pipeline play at a high speed (seems to depend on the speed that
ffdec_h264 decodes), but the speed is not controllable. In a reduced
pipeline (see below) everything works. Therefore for me the
multithreading problem is more or less solved by using this pipeline:
filesrc location=/path/to/my.mp4 ! qtdemux ! h264parse ! ffdec_h264
name=dec ! autovideosink sync=false
It might have been the deinterlace or ffmpegcolorspace that caused
ffdec_h264 to decode only with a single thread...
[1]
http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+13%3A+Playback+speed
[2] http://archive.org/details/ThePhantomoftheOpera
On 12/13/2012 01:35 PM, Peter Rennert wrote:
> Although in my code it does not change anything of the behaviour, I
> just found the theoretical solution to my idea of skipping frames for
> faster playback:
>
> "When doing fast forward (rate > 1.0) or fast reverse (rate < -1.0)
> trickmode playback, the GST_SEEK_FLAG_SKIP flag can be used to
> instruct decoders and demuxers to adjust the playback rate by skipping
> frames. This can improve performance and decrease CPU usage because
> not all frames need to be decoded." [1]
>
> [1]
> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstEvent.html#GstSeekFlags
>
>
> On 12/13/2012 01:08 PM, Peter Rennert wrote:
>> Thanks Juraj these were good tips.
>>
>> I needed to add a queue before ffdec_h264, but there must not be one
>> after ffdec_264. Furthermore, I needed to set sync=false at the image
>> sink. However, now it seems as the filesink (or qtdemux/h264parse?)
>> is not feeding the data quickly enough. (I am just at x2 speed, my
>> disk I/O maximum should far ahead). My ffdec_264 sends now the
>> following warning message when I try to increase the speed:
>>
>> 0:00:06.193160955 3058 0x27f08f0 WARN videodecoder
>> gstvideodecoder.c:2847:gst_video_decoder_alloc_output_frame:<dec>
>> failed to get buffer wrong-state
>> 0:00:06.193239481 3058 0x27f08f0 WARN ffmpeg
>> gstffmpegviddec.c:1218:gst_ffmpegviddec_frame:<dec> ffdec_h264:
>> decoding error (len: -1, have_data: 0)
>>
>> I got rid of those warnings when I add a queue after the filesink.
>>
>> Still there are problems with my pipeline:
>>
>> First problem is that the standard speed seems to be twice as normal.
>>
>> Second, although I see that the computations are now spread over
>> several cores and I do not get anymore warnings from ffdec_h264 after
>> keyPress (and respective the seek event), the playback speed does not
>> change at all anymore.
>>
>> I put my test code below. Its in python (my first attempt of python
>> and gstreamer), but I think it should be readable for people who use
>> C or C++, too. I suspect I might use wrong flags in
>> gst.event_new_seek().
>>
>> P.S.: just to make it complete for documentation purposes, playing a
>> bit around with it and using
>>
>> self.pipeline.send_event(gst.event_new_seek(self.pbRate, gst.FORMAT_TIME,
>> gst.SEEK_FLAG_FLUSH, gst.SEEK_TYPE_NONE,
>> gst.CLOCK_TIME_NONE,
>> gst.SEEK_TYPE_NONE, gst.CLOCK_TIME_NONE))
>>
>> instead of
>>
>> self.pipeline.send_event(gst.event_new_seek(self.pbRate, gst.FORMAT_TIME,
>> gst.SEEK_FLAG_FLUSH, gst.SEEK_TYPE_NONE, 0,
>> gst.SEEK_TYPE_NONE, 0))
>>
>> gives me a python error:
>>
>> Traceback (most recent call last):
>> File "pyGstViewer.py", line 61, in keyPress
>> self.increasePlaybackSpeed()
>> File "pyGstViewer.py", line 70, in increasePlaybackSpeed
>> gst.SEEK_TYPE_NONE, gst.CLOCK_TIME_NONE))
>>
>>
>> ===== CODE =====
>>
>> import sys, os
>> import pygst
>> pygst.require("0.10")
>> import gst
>> import pygtk, gobject
>> import gtk
>>
>> class GTK_Main:
>> def __init__(self):
>> self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
>> self.window.connect("destroy", gtk.main_quit, "WM destroy")
>> vbox = gtk.VBox()
>> self.window.add(vbox)
>> vbox.pack_start(gtk.Label("Please type text"))
>> entry = gtk.Entry()
>> vbox.pack_start(entry)
>> self.window.connect("key-press-event", self.keyPress)
>>
>> self.window.show_all()
>>
>> self.pipeline = gst.parse_launch("filesrc " +
>> "location=/home/peter/vid/20121207/00/2012-12-07.00-00-00.mp4 ! " +
>> "queue ! qtdemux ! h264parse ! queue ! ffdec_h264
>> name=dec ! " +
>> "ffmpegcolorspace ! deinterlace ! xvimagesink sync=false")
>>
>> self.dec = self.pipeline.get_by_name("dec")
>>
>> self.dec.set_property("max-threads", 8);
>>
>> self.pbRate = 1
>>
>> self.pipeline.set_state(gst.STATE_PLAYING)
>>
>> bus = self.pipeline.get_bus()
>> bus.add_signal_watch()
>> bus.connect("message", self.onMessage)
>>
>>
>> def keyPress(self, widget, event):
>> print "keypress event!!"
>> key = gtk.gdk.keyval_name(event.keyval)
>> if key == "d":
>> self.increasePlaybackSpeed()
>> else:
>> self.decreasePlaybackSpeed()
>>
>> def increasePlaybackSpeed(self):
>> print str(self.pbRate) + " --> " + str(self.pbRate + 4)
>> self.pbRate += 4
>> self.pipeline.send_event(gst.event_new_seek(self.pbRate, gst.FORMAT_TIME,
>> gst.SEEK_FLAG_FLUSH, gst.SEEK_TYPE_NONE, 0,
>> gst.SEEK_TYPE_NONE, 0))
>> print "increased playback speed!!!"
>>
>> def decreasePlaybackSpeed(self):
>> self.pbRate -= 2
>> self.pipeline.send_event(gst.event_new_seek(self.pbRate, gst.FORMAT_TIME,
>> gst.SEEK_FLAG_FLUSH, gst.SEEK_TYPE_NONE, 0,
>> gst.SEEK_TYPE_NONE, 0))
>> print "decreased playback speed!!!"
>>
>> def onMessage(self, bus, message):
>> t = message.type
>> if t == gst.MESSAGE_EOS:
>> self.pipeline.set_state(gst.STATE_NULL)
>> elif t == gst.MESSAGE_ERROR:
>> self.pipeline.set_state(gst.STATE_NULL)
>> err, debug = message.parse_error()
>> print "Error: %s" % err, debug
>>
>> GTK_Main()
>> gtk.gdk.threads_init()
>> gtk.main()
>>
>>
>>
>> On 12/13/2012 11:18 AM, Juraj Holtak wrote:
>>>
>>> And maybe the xvimagesink should have sync=false set too... maybe...
>>>
>>> On Dec 13, 2012 12:15 PM, "Juraj Holtak" <juraj.holtak at gmail.com
>>> <mailto:juraj.holtak at gmail.com>> wrote:
>>>
>>> Hi,
>>>
>>> Maybe worth a try:
>>>
>>> Put an "queue" element before and after ffdec_h264 and use
>>> max-threads=<your_cpu_count>. I imagine it worked like this for
>>> me but maybe I just had luck...
>>>
>>> Juraj
>>>
>>> On Dec 13, 2012 2:03 AM, "Peter Rennert" <p.rennert at cs.ucl.ac.uk
>>> <mailto:p.rennert at cs.ucl.ac.uk>> wrote:
>>>
>>> Dear all,
>>>
>>> I want to increase the playback of a video quite
>>> drastically. At the moment I am playing a test video
>>> sequence with
>>>
>>> gst-launch-0.10 filesrc location=/path/to/my.mp4 ! qtdemux !
>>> h264parse ! ffdec_h264 name=dec ! ffmpegcolorspace !
>>> deinterlace ! xvimagesink
>>>
>>> and at some point the video speed cannot get increased
>>> further, because it uses only one CPU core to decode the
>>> video. Gstreamer starts complaining of getting behind the
>>> timestamps of the stream and crashes.
>>>
>>> I could think of two solutions:
>>> - Is there a "native" way of making use of more than
>>> just a single core to decode the video frames? I tried to
>>> set "max-threads" of ffdec_h264 to 4, but it still only uses
>>> a single core.
>>>
>>> or,
>>> - Is there a way of skipping frames? As I need only a
>>> "effective" framerate on the screen of about 25fps I could
>>> just decode the frames I need and skip the other frames.
>>> Then I would not need more CPU power than for realtime playback.
>>>
>>> or.
>>> - Is there a way to split the stream after the qtdemux
>>> and use several decoders in a kind of decoder pool
>>> (distribute the frames between them) and unify the stream after?
>>>
>>> My videos are in H264 format. I am not sure if the
>>> non-keyframes are encoded with respect to the previous frame
>>> or with respect to the last key-frame. In that case I could
>>> try to filter the key-frames and send them to every decoder
>>> in the pool and decode the any frame with that as reference
>>> until the next key-frame. With some management I could try
>>> to merge the output of the pool decoder with a
>>> input-selector. But that sounds bulky :\ So if one knows how
>>> to get to one of the first two solutions, I would be very
>>> happy...
>>>
>>>
>>> Cheers,
>>>
>>> Peter
>>>
>>> PS I am happy to provide an example video. Maybe something
>>> is wrong with the encoding there that prevents ffdec_h264 to
>>> use multiple cores
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> <mailto:gstreamer-devel at lists.freedesktop.org>
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>
>>>
>>>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20121213/5d5f8473/attachment-0001.html>
More information about the gstreamer-devel
mailing list