Adding typefind function for a new Gst Type, "tensors"

Mon Jul 23 17:15:09 UTC 2018

I apologize for all the emails.  I botched something else:  I had incorrect
comments around the `ptr` argument.

Should be:

// Executes typefind logic for "cust/om" media type
// @param ptr  An optional pointer to userdata
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)

On Mon, Jul 23, 2018 at 10:00 AM, David Ing <ding at panopto.com> wrote:

> Sorry, I borked that code example a bit.
>
> The typefind function should be more like this
>
> // Executes typefind logic for "cust/om" media type
> // @param ptr  A pointer to the first byte of the stream.
> static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
> {
>     int64_t offset = 0;
>     uint32_t size = 100;
>     const guint8 *data = gst_type_find_peek(typeFind, offset, size);
>
>     if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
>     {
>         // extract other data from the stream so you can feed it into the caps
>         // replace the `...` in the next statement with the caps data.  See documentation
>         // for `gst_caps_new_simple` for more information.
>
>         GstCaps* caps = gst_caps_new_simple("cust/om", ...);
>         gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, caps);
>         gst_caps_unref(caps);
>     }
> }
>
>
>
> On Mon, Jul 23, 2018 at 9:36 AM, David Ing <ding at panopto.com> wrote:
>
>> I do not have an answer to your questions, but I noticed a documentation
>> page:
>>
>> https://gstreamer.freedesktop.org/documentation/plugin-devel
>> opment/advanced/media-types.html#table-of-other-types
>>
>>
>> At the very bottom of that page, I noticed an empty table "Table of Other
>> Types" which seems to suggest that you can define any type you want (with
>> any prefix, not just "audio/" or "video/").
>>
>> I am guessing that you would simply write a plugin with a typefind
>> function like this ...
>>
>>
>> // Executes typefind logic for "cust/om" media type
>> // @param ptr  A pointer to the first byte of the stream.
>> static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
>> {
>>     int64_t offset = 0;
>>     uint32_t size = 100;
>>     const guint8 *data = gst_type_find_peek(typeFind, offset, size);
>>
>>     if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
>>     {
>>         GstCaps* caps = ...
>>         gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, gst_caps_new_simple("cust/om", caps));
>>         gst_caps_unref(caps);
>>     }
>> }
>>
>> And in your `plugin_init` method, you would need to register the typefind
>> function using `gst_type_find_register`.
>>
>> Notice that in the typefind function, you can define the caps of the
>> stream (such as the dimensionality of your tensor).  I believe that the
>> caps structure can contain any fields you want.
>>
>> Next your plugin should define a pad template having SINK caps which are
>> compatible with the caps that would be detected by your typefind function
>> .  Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ...
>> but it needs to contain the code which transforms the stream from SINK to
>> SRC.
>>
>> I can't see any reason why that wouldn't work, but I am just a novice:
>> Someone should correct me if I am wrong.
>>
>> On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <myungjoo.ham at samsung.com>
>> wrote:
>>
>>> Dear Gstreamer Developers,
>>>
>>>
>>> We have been working on developing Gstreamer plugins to handle
>>> general neural network models implemented with on-the-shelf frameworks
>>> (such as Tensorflow or Caffe) as media filters.
>>>
>>>
>>> I have a few questions on how to register a new custom "typefind" for
>>> a new non-media type, "tensor/tensors".
>>>
>>>
>>>
>>> Because the "sources" are general media streams, such as "camera", "mic",
>>> or "media files", the to-be-proposed type, "other/tensor" and
>>> "other/tensors", do not have their file format, yet.
>>> Their meta data are handled as pad-capabilities and
>>> the buffer between elements contains the tensor data only
>>> (similar to what video/x-raw does)
>>>
>>> However, assuming that we need to register "typefind",
>>> later to be considered to become "good plugin" some day,
>>> it appears that we need to define headers as if they are
>>> stored as files. (for debugging and testing purpose, we may need this
>>> anyway)
>>>
>>>
>>>
>>> Q1. Is it ok to define "other/tensor(s)" type to have some headers
>>> (including dimensions of tensors) when it is stored in file and
>>> not to have any meta data in inter-gst-element transfer (like
>>> video/x-raw)?
>>>
>>> Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
>>> the saved file format for tensors and use "other/tensor(s)" for
>>> video/x-raw-like handling?
>>>
>>> Q3. Or is it ok not to register any and possible to upstream to
>>> gst-plugins-good with these new media types (other/tensor and
>>> other/tensors)?
>>>
>>>
>>>
>>> As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
>>> this project for a while and returned), we've finally got internal
>>> approval
>>> to fully open sources (In LGPL-2.1.)
>>> We will be migrating the code to github.com/nnsuite soon, probably in
>>> this week.
>>>
>>>
>>> The full migration requires CI/CD systems to be migrated as well,
>>> so the migration of developing process might take some time.
>>>
>>>
>>>
>>> We hope, someday, we can upstream the plugins to gst-plugins-good. :)
>>>
>>>
>>>
>>>
>>>
>>> Cheers,
>>> MyungJoo
>>>
>>> --
>>>
>>> MyungJoo Ham (함명주), Ph.D.
>>> Autonomous Machine Lab., AI Center, Samsung Research.
>>> Cell: +82-10-6714-2858
>>>
>>>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180723/2b8c688b/attachment-0001.html>