IMX6 : Video streaming over serial port using Gstreamer and H264 encoder

[James Cameron]

On Mon, Jun 25, 2018 at 05:39:42AM -0700, vk_gst wrote:
> Hello GStreamer Experts,

I'm a serial port expert.  ;-)

> Recently, I have started to work on a project that aims at live
> video streaming applications based on imx6 processors. A quick
> description of what I have done so far and what I am trying to do:
> 
> 1. Setup: imx6 board(Boundary devices Sabre Lite) acting as the
> video server(using GStreamer imx plugins), and a PC running Ubuntu
> on it that receives data from imx6 and streams the video using
> GStreamer features.
> 2. On the imx6 processor, I run the 'testvideosrc' and have
> successfully streamed it using the RTP over UDP.
> 3. Since accessing the serial port is similar to writing and reading
> a file in Linux,

Yes, to a certain extent.  There are some differences in behaviour;
they have a write rate in bits per second bounded by the baud rate,
and their read rate is bounded by the baud rate.  Transmit and receive
buffers change the rate in the very short term.  When an application
hits the rate limit, I/O is stalled.

> next I tried out dumping the video data from imx6 board to a serial
> port and reading this serial port on the PC. For this, the baud rate
> of both devices was configured to 115200 baud.

You might test the effective baud rates of each port are compatible,
that the drivers are implemented correctly, and that the electrical
edges of the signals are intact and slewing correctly.  Prepare a
large test set of random numbers; about 30 seconds worth.  Configure
the ports for raw data using stty, if not already done.  Use pv or cat
to send and receive the test set.  Use a binary comparison program to
look for changes or gaps.

> The encoding 'bitrate' is configured to 5Kbps. Here are the
> commands:
> 
> imx6: gst-launch-1.0 -v videotestsrc pattern=18 !
> video/x-raw,width=100,height=50 ! imxvpuenc_h264 bitrate=5 ! h264parse !
> filesink location=/dev/ttyUSB1 
> 
> PC: CAPS=video/x-h264
> PC:  gst-launch-1.0 -v filesrc location=/dev/ttyUSB1 ! $CAPS ! h264parse !
> avdec_h264 ! autovideosink sync=true
> 
> There are no errors observed on the imx6 board. 
> However, I see the following errors at the PC side : 
> PC:# GST_DEBUG=3 gst-launch-1.0 -v filesrc location=/dev/ttyUSB1 ! $CAPS !
> h264parse ! avdec_h264 ! autovideosink sync=true
>  Setting pipeline to PAUSED …
> 0:00:00.066439392 15475 0x556d8a01d160 WARN basesrc
> gstbasesrc.c:3583:gst_base_src_start_complete: pad not activated yet
>  Pipeline is PREROLLING …
> 0:00:21.730466251 15475 0x556d8a000940 WARN capsfilter
> gstcapsfilter.c:455:gst_capsfilter_prepare_buf: error: Filter caps do not
> completely specify the output format
>  0:00:21.730523691 15475 0x556d8a000940 WARN capsfilter
> gstcapsfilter.c:455:gst_capsfilter_prepare_buf: error: Output caps are
> unfixed: video/x-h264, width=(int)[ 1, 8192 ], height=(int)[ 1, 8192 ],
> framerate=(fraction)[ 0/1, 2147483647/1 ]
>  0:00:21.730676173 15475 0x556d8a000940 WARN basetransform
> gstbasetransform.c:2159:default_generate_output: could not get buffer from
> pool: error
>  0:00:21.730742223 15475 0x556d8a000940 WARN basesrc
> gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
>  0:00:21.730775478 15475 0x556d8a000940 WARN basesrc
> gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason error
> (-5)
>  ERROR: from element /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
> Filter caps do not completely specify the output format
>  Additional debug info:
>  gstcapsfilter.c(455): gst_capsfilter_prepare_buf ():
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
>  Output caps are unfixed: video/x-h264, width=(int)[ 1, 8192 ],
> height=(int)[ 1, 8192 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
>  ERROR: pipeline doesn’t want to preroll.
>  Setting pipeline to NULL …
> Freeing pipeline …

Does this also occur if the data is saved to a file on the imx6 and
then read from file on the PC?

> 4. Since, the encoding rate is 5Kbps (bitrate=5, as specified in the
> above command), I suppose it is possible to send this amount of data
> via the serial port.

Yes, as it should be less than 115200 baud.  But you could also prove
it by counting bytes over a measured time.

> I realize that currently, the caps negotiation fails, however,
> I am unsure of how to proceed with this.

I've no suggestion on that.

> 5. On the PC side, reading the serial port with 'cat /dev/ttyUSB1'
> succeeds with limited data. The data is unreadable(as expected),
> however it is not a continuous stream.

It should not be continuous.  Your imx6 pipeline constrains the
bitrate of encoding output 5000 bits per second, and 115200 bits per
second is a faster clock, therefore the data will come in short
bursts.  The serial link will spend time idle.

> Does anyone have an idea, on how to solve this.  My later test,
> would be to use an actual camera(MIPI) and try streaming it over
> serial port. Does it seem feasible or is a completely crazy idea to
> do?

Properly working serial ports can transmit and receive digital video;
but I'm not confident that GStreamer elements can handle the unusual
time-based behaviours of serial ports, as compared to files.  I'd hope
they can, but I've not tried it.

-- 
James Cameron
http://quozl.netrek.org/