[RFC] wl_surface video protocol extension

[James Courtier-Dutton]

Hi,

I have extensive experience with video streaming and display having worked
on the xine video player.
Setting time stamps on a stream of video frames is not the entire solution.
The biggest problem with displaying video on computer displays is "frame
rate adaption".
I.e. The frame rate of the video frames might not equal the frame rate of
the display.
What is really needed is a predictor.
I.e. If I send this frame to the video hardware, what is the exact time it
will be displayed.
A predictor might be difficult, so some callback after the frame is
displayed giving the exact time the frame was displayed would also help.
The video player could then use this information to adjust the frame
submission to ensure the frame was displayed closer to when it needs to be.

There are various artifacts that appear as a result of frame rate adaption.
One of these is field dominance. This is where the top field is displayed
for longer than the bottom field. This results in the appearance of
flickering lines to the viewer.
If you could determine how long the top field was presented to the user,
you could maybe make adjustments to make the bottom field appear for a
similar amount of time.

So, I would propose the following.
1) sequence number the frames at the display rate.
2) provide an api for the application to predict the time of each frame
number in the sequence. I.e The next 5 frames will display at time
X1,X2,X3,X4,X5. Also provide a way to determine the "frame submited to
frame displayed" latency. So the user application knows how many frames in
advance it needs to do the submit.
3) provide an api for the application to ensure that a frame it submits
will get displayed at display rate sequence number X.
4) If frame X has already past, only display it if there are now newer
frames in the quene, otherwise drop it.

Summary:
For the best video streaming experience, timestamps on the frame are not
very useful, what is useful is predicting exactly when the frame will
actually be displayed to the user.

Kind Regards

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20131017/66b75bd1/attachment.html>