[gst-devel] editing list concept - still not sure
Ronald Bultje
rbultje at ronald.bitfreak.net
Fri May 11 14:37:12 CEST 2001
Hi everyone,
for the MJPEGtools, I'm currently thinking of making something like a
concept of editing lists for video editing. We can already do things like
layer-combining, watermarking, adding a logo or text to the video,
blablabla, all very interesting, but how do we make an editlist for this?
We do have a concept for this but it doesn't work very well (that's my
opinion :) ), basically it can only do one transformation at a time (so
either you do scene transitions or you show the logo in the top right
corner, but you can't do both in one single editing list, you need
multiple editing lists for that.
And I don't like that ;-)
Since we're gonna integrate some parts of us into gstreamer anyway sooner
or later, I'd like to think of a new way of editing list
concepts and hoping that you have some good ideas, or maybe you have
already thought of a concept. We can then use the same concept which would
make switching between the two easier (plus, we would have compatibility).
Basically, I suppose that you want to use *any* input movie, get *any*
arbitrary frame from that input movie on request and do *any* possible
effect or possible combination of effects on one or more inputs and get a
single resulting output stream. And it should be easy for the NLE ;-)
Our editlists look, then, like this (pipelists, not editing lists since
they are *really* old):
-------list.pli---------
LAV PipeList #pli file
PAL #pal/ntsc
4 #num of input
lav:/tmp/movie1.avi #input 0 (275 frames)
lav:/home/me/movie2.avi #input 1 (275 frames)
yuv:/home/me/yuv/image_overlay.yuv #input 2
yuv:/home/me/yuv/image_transparency.yuv #input 3
225 1 #length, num of inputs
0 0 #input0, start from frame 0
- #no transformation
50 2 #length, num of inputs
0 225 #input0, from frame 225
1 0 #input1, start from frame 0
transist.flt 0 255 #use transist.flt as effect
225 3 #length, num of inputs
1 50 #input1, start from frame 50
2 0 #input2
3 0 #input3
matteblend.flt #effect
-------------------------------
(all #....) stuff is not really in the editing list but it's just to clear
it up a bit)
the yuv is an image, lav is an animation. transist.flt = scene transition,
matteblend.flt = for adding yuv data (input2) overlayed on frames of
input1, according to the black-white data of image3 (black = paint, white =
don't paint (since YUV has no transparency).
You see the double effects are not really possible but that wasn't a
problem when we started this but it'd be nice of thinking of a way to do it
since technically, we can od it the problem is just the editing lists
themselves.
You could for example think of this, for enabling double effects:
-------------------------------------------------
LAV PipeList #pli file
PAL #pal/ntsc
4 #num of input
lav:/tmp/movie1.avi #input 0 (275 frames)
lav:/home/me/movie2.avi #input 1 (275 frames)
yuv:/home/me/yuv/image_overlay.yuv #input 2
yuv:/home/me/yuv/image_transparency.yuv #input 3
500 3 #length, num of inputs
225 1 #' ' means sub-level
0 0 #input0, start from frame 0
- #no effect
+50 2 #'+' means same sequence
0 225 #input0, from frame 225
1 0 #input1, start from frame 0
transist.flt 0 255 # use transist.flt as effect
+225 1 #sublevel/same sequence
1 50 #input1, start from frame 50
- #no effect
2 0 #second argument for main sequence
3 0 #third argument for main sequence
matteblend.flt #effect
-------------------------------------------------
This would mean that you start with input0, then a transition between
input0/input1 and then input 1, and over the whole of this, you add an
image (I sent this one to mjpeg-developers@ a while ago). The space means
that it is a 'sub-transformation', so you can then in theory have an
unlimited number of transformations. The '+' means that it is not a new
argument but still a part of the previous argument. So transist.flt over
video1+video2 and video3 means a transformation with subsequently video1
and video2 as first input and video3 as second input. Tansist.flt over
video1 and video2+video3 means that video1 is the first input and video2
and video3 together are the second input.
This one also has severe drawbacks, though:
if you want to show the yuv-image one second after input0 starts (so first
5 frames without title), that becomes quite hard. You need to make a
25-frame entry for input 0 alone and the continue the rest of the
video-animation with a new entry referring to input0 again. That's quite
annoying, especially if you want the title to appear somewhere halfway the
transition.
Especially if you want to make this an editinglist for a NLE, this is a
major disadvantage. Because how is the NLE gonna tell that the first entry
for input0 and the second entry for input0/input[01]-transition are the
same? You definately want the video editor to know that they both belong to
the same scene. I'd rather keep the NLE dumb and the editing list smart (or
both smart - but the editing list should be smart first!).
So I guess that in theory, we need to make something for editing lists with
the properties:
*) it needs to have only *one* entry for each video scene, movie or
whatever. Double entries make it difficult for the NLE and thus should be
prevented. Each entry represents not a movie, but a scene.
*) making multiple transformations at the same time possible
One way of doing this would be to add time/frame-no.-stamping to each
entry.
I can't think of other good ways...
In that case, you tell for each entry at which frame of the total animation
it starts.
To make multiple transformations of the movie possible, you probably need
to stamp it with sequence numbers as well... In total, this would become
kind of a mess - in my mind, that is. So how far are you guys with the
concept of editing lists and how would you suggest top make it look like?
For example, you could have something like:
-----------------------------
EDITLIST
{
INPUTS 3
{
input0 = open_video("/tmp/movie1")
input1 = open_video("/tmp/movie2")
input2 = open_image("/tmp/image.png")
}
SEQUENCES 2
{
sequence0 = sequence_from_video(input0, 0, 275)
sequence1 = sequence_from_video(input1, 0, 275)
}
RESULT 5
{
r0 = result_from_sequence(sequence0, 0, 225)
r1 = scene_transition(simple, sequence0, 225, sequence2, 0, 50)
r2 = result_from_sequence(sequence1, 50, 225)
r3 = result_from_result(r0+r1+r2, 0, 50)
r4 = show_image_over_video(r0+r1+r2, 50, 450, input2)
}
show(r3+r4)
}
-----------------------------
inputX would indicate what the input movies are, sequenceX indicates
individual scenes (good for the NLE).
sequence_from_video(input, x, l) makes a sequence of input 'input starting
at frame 'x' with length 'l'
result_from_sequence(sequence, x, l) = get simple result from sequence
'sequence' of length 'l' starting at frame 'x'
scene_transition(type, sequence1, x1, sequence2, x2, l) makes a transition
of type 'type' with length 'l' between sequence1 starting at frame x1 and
sequence2 starting at frame x2.
result_from_result(result(s), x, l) makes a new result out of existing
results r1+r2+r3, first 50 frames, starting at frame 0 of the total.
show_image_over_video(result(s), x, l, image) shows image 'image' (an
input) over the results 'result(s)', starting at frame x with length l.
show(result(s)) = endresult, what to show to the end user.
Sequences are the scenes for the NLE, inputs are the inputs for
gstreamer/mjpegtools, resulsts is the calculation work to get the preferred
outputstream and show() is the resulting outputstream.
It would still be quite some work to get the NLE to know what exactly is
going in this editing list but I guess this comes closer to what the
perfect editing list should be like.
Well, these are just ideas, who has a good idea for 'the perfect editlist'?
Awaiting your replies,
Ronald Bultje
--
---------------------------------------------------.
-- .-. | Ronald Bultje |
-- /V\ | Running: Linux 2.4.4 and OpenBSD 2.8 |
-- // \\ | E-mail : rbultje at ronald.bitfreak.net |
-- /( )\ | WWW : http://ronald.bitfreak.net/ |
-- ^^-^^ | *** Warning: Unix Addicted *** |
---------------------------------------------------'
More information about the gstreamer-devel
mailing list