[gst-devel] Proposition for a media test suite

[Ronald S. Bultje]

On Mon, 2005-11-14 at 10:56 +0100, Edward Hervey wrote:
>   I put most of the main lines on gnome live here :
> http://live.gnome.org/GStreamerMediaTest

- not everybody speaks or wants to speak python. Everybody in our team
speaks C (or is supposed to).
- it may be nice to be able to randomly add files that we fix in
bugzilla. Make sure adding files is easy for maintainers.
- possibly extra tests for metadata, video snapshotting, stream
properties etc. (see the nautilus property pages, thumbnailer etc. in
totem). Also makes for nice extra visual effects in the status page for
the testsuite (people love this).
- it has never been clear to me how such things are to be handled in a
non-playback fashion. Is a seek event in GST_STATE_NULL allowed? I
assume that we want to be able to set a pipeline to PAUSED but not get
any frames until after we've done a seek (for editing purposes).
Problem: how do we handle decodebin-pipelines then (where pipeline setup
is equal to preroll because it's the same statechange)? Once we start
doing this kind of stuff in testsuites, we want such things to be
thought out first.
- what is the difference between SEEK_FLAG_KEYFRAME and no seek flags
for the visual user perception anyway? How do we test and judge
compliance? Is 0.1s off on the next buffer for _EXACT wrong? For 1fps
avi? For 96kHz mp3? And so on.
- Should plugins who don't implement any special seeking mode ignore
seeking flags? Audio plugins can probably ignore _KEYFRAME, but should
they error out if they only match the nearest frame while _EXACT is
given?
- how do we test output? Not all formats are bitexact. Not all formats
give the same output depending on library version, arch/os/compiler (and
this is not a bug!), optimizations used (this is not a bug either!),
etc.; so we cannot use md5s to test, or we'd need whole series of md5s.
- Also test ffmpegcolorspace behaviour on all supported colourspaces
(although this should probably be done in make check).

And after reading the l.g.o page:
- what is the relation between filler events and
GST_BUFFER_FLAG_DISCONT? How's the filler story going anyway?

Ronald