<div dir="ltr">Hello,<div><br></div><div>I have some code for muxing klv and video (included at the end of this email).</div><div><br></div><div>This code is a proof of concept to see if I am doing things correctly.  I have an mpg video with klv and video streams.  I first dumped the klv into a file using the following pipeline:</div><div><br></div><div>gst-launch-1.0 filesrc=input.ts ! tsdemux ! meta/x-klv ! filesink location=klv.bin</div><div><br></div><div>By inspecting the klv, I determined how many klv packets are in the stream and the klv frame rate.</div><div><br></div><div>I then extracted the video frames (using ffmpeg) into a folder.  I also used ffprobe to determine things like the video frame rate.</div><div><br></div><div>What I attempted to do is remux the video frames and klv and end up with something identical to the original file.  The code mostly works though there are some minor differences that I am curious about.</div><div><br></div><div>This code uses the python bindings.  I used <a href="http://gstreamer-devel.966125.n4.nabble.com/Example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-GStreamer-1-8-3-td4684782.html">http://gstreamer-devel.966125.n4.nabble.com/Example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-GStreamer-1-8-3-td4684782.html</a> and <a href="https://github.com/tamaggo/gstreamer-examples/blob/master/test_gst_appsrc_testvideo_mp4mux.py">https://github.com/tamaggo/gstreamer-examples/blob/master/test_gst_appsrc_testvideo_mp4mux.py</a> as guidelines.<br></div><div><br></div><div>One difference I don't understand, is the ffprobe output of the original file has:</div><div><br></div><div>$ ffprobe input.ts<br>[...]</div><div>  Duration: 00:01:05.15, start: 271.125322, bitrate: 5535 kb/s<br></div><div>[...]</div><div><br></div><div>Whereas the ffprobe output of the remuxed file has:</div><div><br></div><div>$ ffprobe remuxed.ts</div><div>[...]</div><div>  Duration: 00:01:04.57, start: 3600.033367, bitrate: 1745 kb/s<br></div><div>[...]</div><div><br></div><div>Note the difference in start time.  What controls the start time here?  I tried setting the start-time property of x264enc, but that didn't do the trick.</div><div><br></div><div>Is there anything in this code that jumps out as the wrong way to do things, or is there a better way to mux together video and klv streams?</div><div><br></div><div>Thanks</div><div><br></div><div>Code:<br></div><div><br></div><div>import matplotlib.pyplot as plt<br>import os<br>import sys<br>import gi<br>import json<br><br>gi.require_version('Gst', '1.0')<br>from gi.repository import Gst, GObject<br><br><br>if len(sys.argv) != 2:<br>    print("Usage: python muxklv.py [config file]")<br>    sys.exit(1)<br><br>config_file = sys.argv[1]<br>fh = open(config_file)<br>config_json = json.load(fh)<br>fh.close()<br><br><br><br>vid_frames_folder = config_json["vid_frames_folder"]<br>vid_frame_rate = config_json["vid_frame_rate"]<br>vid_width = config_json["vid_width"]<br>vid_height = config_json["vid_height"]<br>klv_file = config_json["klv_file"]<br>klv_frame_rate = config_json["klv_frame_rate"]<br>klv_packet_size = config_json["klv_packet_size"]<br>out_file = config_json["out_file"]<br><br><br>GObject.threads_init()<br>Gst.init(None)<br><br># video source elements<br>vsrc = Gst.ElementFactory.make("appsrc", "vidsrc")<br>vqueue = Gst.ElementFactory.make("queue")<br>vtee = Gst.ElementFactory.make("tee")<br><br># klv source elements<br>appsrc = Gst.ElementFactory.make("appsrc")<br>queue_klv = Gst.ElementFactory.make("queue")<br><br># display elements<br>queue_display = Gst.ElementFactory.make("queue")<br>vcvt = Gst.ElementFactory.make("videoconvert", "vidcvt")<br>vsink = Gst.ElementFactory.make("autovideosink", "vidsink")<br><br># recording elements<br>queue_record = Gst.ElementFactory.make("queue")<br>vcvt_encoder = Gst.ElementFactory.make("videoconvert")<br>encoder = Gst.ElementFactory.make("x264enc")<br>muxer = Gst.ElementFactory.make("mpegtsmux")<br>filesink = Gst.ElementFactory.make("filesink")<br><br># configure video element<br>caps_str = "video/x-raw"<br>caps_str += ",format=(string)RGB,width={},height={}".format(vid_width,vid_height)<br>caps_str += ",framerate={}/1".format(int(vid_frame_rate))<br>vcaps = Gst.Caps.from_string(caps_str)<br>vsrc.set_property("caps", vcaps);<br>vsrc.set_property("format", Gst.Format.TIME)<br><br># configure appsrc element<br>caps_str = "meta/x-klv"<br>caps_str += ",parsed=True"<br>caps = Gst.Caps.from_string(caps_str)<br>appsrc.set_property("caps", caps)<br># appsrc.connect("need-data", klv_need_data)<br>appsrc.set_property("format", Gst.Format.TIME)<br><br># configure encoder<br>encoder.set_property("noise-reduction", 1000)<br>encoder.set_property("threads", 4)<br>encoder.set_property("bitrate", 1755)<br><br># configure filesink<br>filesink.set_property("location", out_file)<br>filesink.set_property("async", 0)<br><br>pipeline = Gst.Pipeline()<br>pipeline.add(vsrc)<br>pipeline.add(vqueue)<br>pipeline.add(vtee)<br>pipeline.add(appsrc)<br>pipeline.add(queue_klv)<br>pipeline.add(queue_display)<br>pipeline.add(vcvt)<br>pipeline.add(vsink)<br>pipeline.add(queue_record)<br>pipeline.add(vcvt_encoder)<br>pipeline.add(encoder)<br>pipeline.add(muxer)<br>pipeline.add(filesink)<br><br># link video elements<br>vsrc.link(vqueue)<br>vqueue.link(vtee)<br><br># link display elements<br>vtee.link(queue_display)<br>queue_display.link(vcvt)<br>vcvt.link(vsink)<br><br># link recording elements<br>vtee.link(queue_record)<br>queue_record.link(vcvt_encoder)<br>vcvt_encoder.link(encoder)<br>encoder.link(muxer)<br>muxer.link(filesink)<br><br># link klv elements<br>appsrc.link(queue_klv)<br>queue_klv.link(muxer)<br><br>pipeline.set_state(Gst.State.PLAYING)<br><br>fh = open(klv_file, 'rb')<br>timestamp = 0<br><br>klv_done = False<br>vid_done = False<br>vid_frame_counter = 0<br>klv_frame_counter = 0<br><br>t = 0<br>last_klv_t = 0<br>last_vid_t = 0<br>while True:<br>    if vid_done and klv_done:<br>        break<br>    if t - last_klv_t >= 1.0 / klv_frame_rate:<br>        if not klv_done:<br>            klv_bytes = fh.read(klv_packet_size)<br>            if klv_bytes:<br>                klvbuf = Gst.Buffer.new_allocate(None, klv_packet_size, None)<br>                klvbuf.fill(0, klv_bytes)<br>                klvbuf.pts = int(t * 1e9)<br>                klvbuf.dts = int(t * 1e9)<br><br>                appsrc.emit("push-buffer", klvbuf)<br>                klv_frame_counter += 1<br>                last_klv_t = t<br>                print("klv {} {}".format(klv_frame_counter, last_klv_t))<br>            else:<br>                klv_done = True<br><br>    if t - last_vid_t >= 1.0 / vid_frame_rate:<br>        if not vid_done:<br>            frame_filename = '%s/%04d.jpg' % (vid_frames_folder, vid_frame_counter + 1)<br>            if os.path.isfile(frame_filename):<br>                vid_frame = plt.imread(frame_filename)<br>                data = vid_frame.tostring()<br>                vidbuf = Gst.Buffer.new_allocate(None, len(data), None)<br>                vidbuf.fill(0, data)<br>                vidbuf.pts = int(t * 1e9)<br>                vidbuf.dts = int(t * 1e9)<br><br>                vsrc.emit("push-buffer", vidbuf)<br>                vid_frame_counter += 1<br>                last_vid_t = t<br>                print("vid {} {}".format(vid_frame_counter, last_vid_t))<br>            else:<br>                vid_done = True<br>                continue<br><br>    t += 0.000001<br>    #print(t)<br><br>vsrc.emit("end-of-stream")<br>appsrc.emit("end-of-stream")<br><br>bus = pipeline.get_bus()<br><br>while True:<br>    msg = bus.poll(Gst.MessageType.ANY, Gst.CLOCK_TIME_NONE)<br>    t = msg.type<br>    if t == Gst.MessageType.EOS:<br>        print("EOS")<br>        break<br>        pipeline.set_state(Gst.State.NULL)<br>    elif t == Gst.MessageType.ERROR:<br>        err, debug = msg.parse_error()<br>        # print("Error: %s" % err, debug)<br>        break<br>    elif t == Gst.MessageType.WARNING:<br>        err, debug = msg.parse_warning()<br>        # print("Warning: %s" % err, debug)<br>    elif t == Gst.MessageType.STATE_CHANGED:<br>        pass<br>    elif t == Gst.MessageType.STREAM_STATUS:<br>        pass<br>    else:<br>        pass<br>        # print(t)<br>        # print("Unknown message: %s" % msg)<br><br>pipeline.set_state(Gst.State.NULL)<br><br>print("Bye")<br></div></div>