Mixing v4l2 sources with compositor
Frédéric Sureau
frederic.sureau at veo-labs.com
Thu May 21 02:43:10 PDT 2015
Hi all!
I am using compositor element from gst-plugins-bad to mix 2 live sources
from v4l2src.
Visually, I get: black frames THEN 1 source THEN 2 sources
After this short initialization, everything is OK and the 2 sources are
synchronized.
I have noticed (using identity) that v4l2src buffers do not start pts at
0:00:00.000000000.
First source starts pts at approximately 140ms which correspond to 7
frames at 50 fps.
This is exactly the number of black frames at the beginning.
In v4l2src, pts is calculated from buffer absolute timestamp (coming
from v4l2 driver) and v4l2src PLAYING state absolute time. The
difference between these 2 absolute times gives pts.
Using videorate just after v4l2src makes pts start at 0, but I wonder if
dropping driver timestamps is a good idea for the synchronization of the
2 sources.
Here is a simple pipeline illustrating the problem:
# gst-launch-1.0 -ev v4l2src device=/dev/v4l/by-path/ipu1-capture
num-buffers=10 ! identity silent=false ! "video/x-raw, format=NV12,
width=1280, height=720, framerate=50/1" ! compositor background=black !
fakesink silent=false
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = event
******* (identity0:sink) E (type: stream-start (10254),
GstEventStreamStart,
stream-id=(string)31ff3376f33e26cc48d904ec759b67b7f5e1d88912d38fccd88eb8d2c6ad5983,
flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)0;) 0x17e4598
WARNING: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed
to get value for control 9963815 on device '/dev/v4l/by-path/ipu1-capture'.
Additional debug info:
/home/fred/Projets/veobox3/poky/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer1.0-plugins-good/git-r0/git/sys/v4l2/v4l2_calls.c(928):
gst_v4l2_get_attribute (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
system error: Invalid argument
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ framerate\=\(fraction\)50/1\,\
width\=\(int\)1280\,\ height\=\(int\)720\,\
interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709"
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = event
******* (identity0:sink) E (type: caps (12814), GstEventCaps,
caps=(GstCaps)"video/x-raw\,\ format\=\(string\)NV12\,\
framerate\=\(fraction\)50/1\,\ width\=\(int\)1280\,\
height\=\(int\)720\,\ interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709";)
0x17e45e0
/GstPipeline:pipeline0/GstIdentity:identity0.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ framerate\=\(fraction\)50/1\,\
width\=\(int\)1280\,\ height\=\(int\)720\,\
interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709"
New clock: GstSystemClock
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ framerate\=\(fraction\)50/1\,\
width\=\(int\)1280\,\ height\=\(int\)720\,\
interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709"
/GstPipeline:pipeline0/GstCompositor:compositor0.GstCompositorPad:sink_0: caps
= "video/x-raw\,\ format\=\(string\)NV12\,\
framerate\=\(fraction\)50/1\,\ width\=\(int\)1280\,\
height\=\(int\)720\,\ interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ framerate\=\(fraction\)50/1\,\
width\=\(int\)1280\,\ height\=\(int\)720\,\
interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709"
/GstPipeline:pipeline0/GstIdentity:identity0.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ framerate\=\(fraction\)50/1\,\
width\=\(int\)1280\,\ height\=\(int\)720\,\
interlace-mode\=\(string\)progressive\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)bt709"
WARNING: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed
to get value for control 9963815 on device '/dev/v4l/by-path/ipu1-capture'.
Additional debug info:
/home/fred/Projets/veobox3/poky/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer1.0-plugins-good/git-r0/git/sys/v4l2/v4l2_calls.c(928):
gst_v4l2_get_attribute (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
system error: Invalid argument
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = event
******* (identity0:sink) E (type: segment (17934), GstEventSegment,
segment=(GstSegment)"GstSegment,
flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1,
applied-rate=(double)1, format=(GstFormat)GST_FORMAT_TIME,
base=(guint64)0, offset=(guint64)0, start=(guint64)0,
stop=(guint64)18446744073709551615, time=(guint64)0,
position=(guint64)0, duration=(guint64)18446744073709551615;";) 0x17e46b8
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.145548617, duration: 0:00:00.020000000, offset: 0,
offset_end: 1, flags: 00000040 discont ) 0x7550e040
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event
******* (fakesink0:sink) E (type: stream-start (10254),
GstEventStreamStart, stream-id=(string)agg-c76d3c61,
flags=(GstStreamFlags)GST_STREAM_FLAG_NONE;) 0x75504450
/GstPipeline:pipeline0/GstCompositor:compositor0.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1280\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)bt709\,\
framerate\=\(fraction\)50/1"
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event
******* (fakesink0:sink) E (type: caps (12814), GstEventCaps,
caps=(GstCaps)"video/x-raw\,\ format\=\(string\)NV12\,\
width\=\(int\)1280\,\ height\=\(int\)720\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)bt709\,\
framerate\=\(fraction\)50/1";) 0x75504498
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1280\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)bt709\,\
framerate\=\(fraction\)50/1"
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event
******* (fakesink0:sink) E (type: segment (17934), GstEventSegment,
segment=(GstSegment)"GstSegment,
flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1,
applied-rate=(double)1, format=(GstFormat)GST_FORMAT_TIME,
base=(guint64)0, offset=(guint64)0, start=(guint64)0,
stop=(guint64)18446744073709551615, time=(guint64)0,
position=(guint64)18446744073709551615,
duration=(guint64)18446744073709551615;";) 0x755044e0
Redistribute latency...
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = preroll
*******
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.000000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x7550e680
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.165558949, duration: 0:00:00.020000000, offset: 1,
offset_end: 2, flags: 00000000 ) 0x7550e0e0
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.020000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05000
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.040000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f050a0
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.060000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05140
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.080000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f051e0
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.100000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05280
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.120000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05320
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.185553949, duration: 0:00:00.020000000, offset: 2,
offset_end: 3, flags: 00000000 ) 0x7550e180
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.140000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f053c0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.205550950, duration: 0:00:00.020000000, offset: 3,
offset_end: 4, flags: 00000000 ) 0x7550e220
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.160000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05460
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.225549949, duration: 0:00:00.020000000, offset: 4,
offset_end: 5, flags: 00000000 ) 0x7550e2c0
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.180000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f053c0
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.200000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05320
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.245550283, duration: 0:00:00.020000000, offset: 5,
offset_end: 6, flags: 00000000 ) 0x7550e360
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.220000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05280
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.240000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f051e0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.265548950, duration: 0:00:00.020000000, offset: 6,
offset_end: 7, flags: 00000000 ) 0x7550e400
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.260000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05140
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.285559283, duration: 0:00:00.020000000, offset: 7,
offset_end: 8, flags: 00000000 ) 0x7550e4a0
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.280000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f050a0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.305560283, duration: 0:00:00.020000000, offset: 8,
offset_end: 9, flags: 00000000 ) 0x7550e040
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.300000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x75f05000
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain
******* (identity0:sink) (1382400 bytes, dts: none,
pts:0:00:00.325547950, duration: 0:00:00.020000000, offset: 9,
offset_end: 10, flags: 00000000 ) 0x7550e0e0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = event
******* (identity0:sink) E (type: eos (28174), ) 0x17e4700
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.320000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x7550e720
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = chain
******* (fakesink0:sink) (1382400 bytes, dts: none, pts:
0:00:00.340000000, duration: 0:00:00.020000000, offset: -1, offset_end:
-1, flags: 00000000 ) 0x7550e680
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event
******* (fakesink0:sink) E (type: eos (28174), ) 0x75504648
Got EOS from element "pipeline0".
Execution ended after 0:00:00.347907666
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Thanks for your help!
Fred
More information about the gstreamer-devel
mailing list