<div dir="ltr">Sorry, I borked that code example a bit.<div><br></div><div>The typefind function should be more like this<div><br></div><div><pre style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:"Courier New";font-size:9pt"><pre style=""><span style="color:rgb(128,128,128);font-style:italic">// Executes typefind logic for "cust/om" media type<br></span><span style="color:rgb(128,128,128);font-style:italic">// @param ptr A pointer to the first byte of the stream.<br></span><span style="color:rgb(0,0,128);font-weight:bold">static void </span>gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)<br>{<br> int64_t offset = <span style="color:rgb(0,0,255)">0</span>;<br> uint32_t size = <span style="color:rgb(0,0,255)">100</span>;<br> <span style="color:rgb(0,0,128);font-weight:bold">const </span>guint8 *data = gst_type_find_peek(typeFind, offset, size);<br><br> <span style="color:rgb(0,0,128);font-weight:bold">if </span>(data && data[<span style="color:rgb(0,0,255)">0</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'C' </span>&& data[<span style="color:rgb(0,0,255)">1</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'U' </span>&& data[<span style="color:rgb(0,0,255)">2</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'S' </span>&& data[<span style="color:rgb(0,0,255)">3</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'T'</span>)<br> {<br> <span style="color:rgb(128,128,128);font-style:italic">// extract other data from the stream so you can feed it into the caps<br></span><span style="color:rgb(128,128,128);font-style:italic"> // replace the `...` in the next statement with the caps data. See documentation<br></span><span style="color:rgb(128,128,128);font-style:italic"> // for `gst_caps_new_simple` for more information.<br></span><span style="color:rgb(128,128,128);font-style:italic"><br></span><span style="color:rgb(128,128,128);font-style:italic"> </span>GstCaps* caps = gst_caps_new_simple(<span style="color:rgb(0,128,0);font-weight:bold">"cust/om"</span>, ...);<br> gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, caps);<br> gst_caps_unref(caps);<br> }<br>}</pre></pre><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 23, 2018 at 9:36 AM, David Ing <span dir="ltr"><<a href="mailto:ding@panopto.com" target="_blank">ding@panopto.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I do not have an answer to your questions, but I noticed a documentation page:<br></div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><a href="https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html#table-of-other-types" target="_blank">https://gstreamer.freedesktop.<wbr>org/documentation/plugin-<wbr>development/advanced/media-<wbr>types.html#table-of-other-<wbr>types</a></div></blockquote><div><div><br></div><div>At the very bottom of that page, I noticed an empty table "Table of Other Types" which seems to suggest that you can define any type you want (with any prefix, not just "audio/" or "video/").</div><div><br></div><div>I am guessing that you would simply write a plugin with a typefind function like this ...</div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><br></div><div><pre style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:"Courier New";font-size:9pt"><pre><pre style="background-color:rgb(255,255,255);font-family:"Courier New";font-size:9pt"><span style="color:rgb(128,128,128);font-style:italic">// Executes typefind logic for "cust/om" media type<br></span><span style="color:rgb(128,128,128);font-style:italic">// @param ptr A pointer to the first byte of the stream.<br></span><span style="color:rgb(0,0,128);font-weight:bold">static void </span>gst_custom_typefind(<wbr>GstTypeFind *typeFind, gpointer ptr)<br>{<br> int64_t offset = <span style="color:rgb(0,0,255)">0</span>;<br> uint32_t size = <span style="color:rgb(0,0,255)">100</span>;<br> <span style="color:rgb(0,0,128);font-weight:bold">const </span>guint8 *data = gst_type_find_peek(typeFind, offset, size);<br><br> <span style="color:rgb(0,0,128);font-weight:bold">if </span>(data && data[<span style="color:rgb(0,0,255)">0</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'C' </span>&& data[<span style="color:rgb(0,0,255)">1</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'U' </span>&& data[<span style="color:rgb(0,0,255)">2</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'S' </span>&& data[<span style="color:rgb(0,0,255)">3</span>] == <span style="color:rgb(0,128,0);font-weight:bold">'T'</span>)<br> {<br> GstCaps* caps = ...<br> gst_type_find_suggest(<wbr>typeFind, GST_TYPE_FIND_MAXIMUM, gst_caps_new_simple(<span style="color:rgb(0,128,0);font-weight:bold">"cust/om"</span>, caps));<br> gst_caps_unref(caps);<br> }<br>}</pre></pre></pre></div></blockquote><div>And in your `plugin_init` method, you would need to register the typefind function using `gst_type_find_register`.</div><div><br></div>Notice that in the typefind function, you can define the caps of the stream
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(such as the dimensionality of your tensor)</span>. I believe that the caps structure can contain any fields you want.</div><div><br></div><div>Next your plugin should define a pad template having SINK caps which are compatible with the caps that would be detected by your typefind function . Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ... but it needs to contain the code which transforms the stream from SINK to SRC.</div></div></div><div><br></div><div>I can't see any reason why that wouldn't work, but I am just a novice: Someone should correct me if I am wrong.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <span dir="ltr"><<a href="mailto:myungjoo.ham@samsung.com" target="_blank">myungjoo.ham@samsung.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Gstreamer Developers,<br>
<br>
<br>
We have been working on developing Gstreamer plugins to handle<br>
general neural network models implemented with on-the-shelf frameworks<br>
(such as Tensorflow or Caffe) as media filters.<br>
<br>
<br>
I have a few questions on how to register a new custom "typefind" for<br>
a new non-media type, "tensor/tensors".<br>
<br>
<br>
<br>
Because the "sources" are general media streams, such as "camera", "mic",<br>
or "media files", the to-be-proposed type, "other/tensor" and<br>
"other/tensors", do not have their file format, yet.<br>
Their meta data are handled as pad-capabilities and<br>
the buffer between elements contains the tensor data only<br>
(similar to what video/x-raw does)<br>
<br>
However, assuming that we need to register "typefind",<br>
later to be considered to become "good plugin" some day,<br>
it appears that we need to define headers as if they are<br>
stored as files. (for debugging and testing purpose, we may need this anyway)<br>
<br>
<br>
<br>
Q1. Is it ok to define "other/tensor(s)" type to have some headers<br>
(including dimensions of tensors) when it is stored in file and<br>
not to have any meta data in inter-gst-element transfer (like video/x-raw)?<br>
<br>
Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent<br>
the saved file format for tensors and use "other/tensor(s)" for<br>
video/x-raw-like handling?<br>
<br>
Q3. Or is it ok not to register any and possible to upstream to<br>
gst-plugins-good with these new media types (other/tensor and other/tensors)?<br>
<br>
<br>
<br>
As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from<br>
this project for a while and returned), we've finally got internal approval<br>
to fully open sources (In LGPL-2.1.)<br>
We will be migrating the code to <a href="http://github.com/nnsuite" rel="noreferrer" target="_blank">github.com/nnsuite</a> soon, probably in this week.<br>
<br>
<br>
The full migration requires CI/CD systems to be migrated as well,<br>
so the migration of developing process might take some time.<br>
<br>
<br>
<br>
We hope, someday, we can upstream the plugins to gst-plugins-good. :)<br>
<br>
<br>
<br>
<br>
<br>
Cheers,<br>
MyungJoo<br>
<br>
--<br>
<br>
MyungJoo Ham (함명주), Ph.D.<br>
Autonomous Machine Lab., AI Center, Samsung Research.<br>
Cell: +82-10-6714-2858<br>
<br>
<br>
______________________________<wbr>_________________<br>
gstreamer-devel mailing list<br>
<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesk<wbr>top.org</a><br>
<a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/<wbr>mailman/listinfo/gstreamer-dev<wbr>el</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>