<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none"></style> </head> <body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"> <p>Hello everyone!<br> </p> <p><br> </p> <p>We have an issue with streaming of H264 encoded video over UDP which comes from an appsrc element<br> </p> <p><br> </p> <p>We need to provide interoperability between a UWP (Windows 10) app and gstreamer.<br> </p> <p>NOTE: we cannot use the <strong>ksvideosrc</strong> element since we're trying to capture the frames from the HoloLens mixed reality camera.<br> </p> <p>For that purpose we are trying to feed the frames as taken from Windows webcam API to an appsrc element, <span style="font-size: 12pt;">encode as h264 and transmit over udp...</span></p> <p>We're using gstreamer 1.12.2 for Windows<br> </p> <p><br> </p> <p>Here it is what I am trying to do:<br> </p> <p><br> </p> <p>Transmitter:</p> <div>AppSrcPipeline.pipeline = gst_parse_launch(</div> <div><span style="white-space:pre"></span>"appsrc name=appsrc_element block=true ! video/x-raw,format=RGB,width=320,height=240,framerate=30/1 ! identity check-imperfect-timestamp=true ! "</div> <div><span style="white-space:pre"></span>"videoconvert ! x264enc ! video/x-h264,profile=\"high-4:4:4\" ! rtph264pay ! udpsink host=192.168.168.98",</div> <div><span style="white-space:pre"></span>&err);<br> </div> <p><br> </p> <p>Receiver:<br> </p> <p>gst-launch-1.0 udpsrc ! application/x-rtp,framerate=30/1 ! rtpjitterbuffer ! rtph264depay ! decodebin ! videoconvert ! ximagesink<br> </p> <p><br> </p> <p>I feed data into the appsrc when the signal "need-data" arrives. The function that feeds the frames has the following:<br> </p> <p><br> </p> <p>void data_feed(GstElement * pipeline, guint size, void *app) {<br> </p> <div><span style="white-space:pre"></span>size_t sz = 3 * 320 * 240 * sizeof(guchar);</div> <div><span style="white-space:pre"></span>GstFlowReturn ret;</div> <div><span style="white-space:pre"></span>static int c = 70;</div> <div><span style="white-space:pre"></span>GstBuffer *buffer = gst_buffer_new_allocate(NULL, sz, NULL);</div> <div><span style="white-space:pre"></span>GstMapInfo info;</div> <div><span style="white-space:pre"></span>static GstClockTime timestamp = 0, duration = 33333333, offset = 0;<br> </div> <div> <div><span style="white-space:pre"></span>gst_buffer_map(buffer, &info, GST_MAP_WRITE);</div> <div><span style="white-space:pre"></span>/*GST_BUFFER_PTS(buffer) = timestamp; // TIMESTAMP<br> </div> <div><span style="white-space:pre"></span>GST_BUFFER_DURATION(buffer) = duration;</div> <div><span style="white-space:pre"></span>GST_BUFFER_OFFSET(buffer) = offset++;</div> <div><span style="white-space:pre"></span>GST_BUFFER_OFFSET_END(buffer) = offset;*/</div> <div><span style="white-space:pre"></span>timestamp += duration;</div> <div><span style="white-space:pre"></span>memset(info.data, c, sz);<span style="white-space:pre"> </span>// for now just feed a grayscale screen with changing intensity<br> </div> <div><span style="white-space:pre"></span>gst_buffer_unmap(buffer, &info);</div> <div><span style="white-space:pre"></span>g_signal_emit_by_name(AppSrcPipeline.src, "push-buffer", buffer, &ret);<br> </div> <div><span style="white-space:pre"></span>gst_buffer_unref(buffer);<br> </div> <div> c++;<br> </div> <div>}<br> </div> <br> </div> <div>So the problem is that only the first frame is ever decoded and displayed on the other side.<br> </div> <div>The transmitter pipeline is in PLAYING state and also there is network traffic, but only one frame is ever drawn.<br> </div> <div><span style="font-size: 12pt;">I have tried different modifications to the pipeline in order to understand which combination of elements works and which doesn't</span></div> <div><span style="font-size: 12pt;"><br> </span></div> <div><span style="font-size: 12pt;">1) replace appsrc -> videotestsrc. Then everything works and I can see the "live" videotest screen on the receiving side</span></div> <div> <div> AppSrcPipeline.pipeline = gst_parse_launch(</div> <div><span style="white-space:pre"></span>"<strong>videotestsrc</strong> name=appsrc_element ! video/x-raw,format=RGB,width=320,height=240,framerate=30/1 ! identity check-imperfect-timestamp=true name=\"identity_element\" ! "</div> <div><span style="white-space:pre"></span>"videoconvert ! x264enc ! video/x-h264,profile=\"high-4:4:4\" ! rtph264pay ! udpsink host=192.168.168.98",</div> <div><span style="white-space:pre"></span>&err);<br> </div> <div><br> </div> <div>2) If I do encoding/decoding but don't "payload" h264 content from the appsrc. The following pipeline also works:<br> </div> <div> <div><span style="white-space:pre"></span></div> <div> <div><span style="white-space:pre"></span>AppSrcPipeline.pipeline = gst_parse_launch(<br> </div> <div><span style="white-space:pre"></span>"appsrc name=appsrc_element block=true ! video/x-raw,format=RGB,width=320,height=240,framerate=30/1 ! identity check-imperfect-timestamp=true ! "</div> <div><span style="white-space:pre"></span>"videoconvert ! x264enc ! video/x-h264,profile=\"high-4:4:4\" ! <strong>decodebin ! videoconvert ! autovideosink</strong> ",</div> <div><span style="white-space:pre"></span>&err);<br> </div> <br> </div> <br> </div> So we can summarize the situation like this:<br> </div> <div>1) appsrc + autovideosink <strong>OK</strong><br> </div> <div>2) videotestsrc + h264 encoding/decoding + RTP streaming <strong>OK</strong><br> </div> <div>3) appsrc + h264 encoding/decoding <strong>OK</strong><br> </div> <div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">4) appsrc + h264 encoding + RTP streaming <strong>PROBLEM</strong></span><br> </div> <div><br> </div> <div>My current understanding of the problem is that I don't provide the correct timestamps on the buffers I produce<br> </div> <div>and this is a problem for the rtph264pay/<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">rtp</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">h264depay elements. I have tried to provide timestamps- if I uncomment</span><br> </div> <div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">the "TIMESTAMP" code block I get an error in the logs and the pipeline doesn't seem to start at all</span></div> <div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">GStreamer-CRITICAL **: gst_segment_to_running_time: assertion 'segment->format == format' failed<br> </span></div> <div>This error message seems to be quite generic so googling it didn't show up any relevant discussions<br> </div> <div><br> </div> <div>Also I have noticed that if I use the combination</div> <div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">appsrc + autovideosink</span></div> <div>the data_feed function is called <span style="font-size: 12pt;">about 20-30 times a second (which corresponds </span><span style="font-size: 12pt;">to</span><span style="font-size: 12pt;"> the desired framerate)</span></div> <div>but if I use <br> </div> <div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">appsrc + h264 encoding + RTP streamin</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">g</span><br> </div> <div>The function is called a couple of hundreds of times per second and utilizes 100% CPU<br> </div> <div><br> </div> <div>It would be much appreciated if anybody can point at possible causes of the problem.<br> </div> <div><br> </div> <div>Best Regards<br> </div> <div>Martin<br> </div> <div><br> </div> <div><br> </div> <div><br> </div> <p><br> </p> </body> </html>