Adding typefind function for a new Gst Type, "tensors"

[David Ing]

Sorry, I borked that code example a bit.

The typefind function should be more like this

// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' &&
data[3] == 'T')
    {
        // extract other data from the stream so you can feed it into the caps
        // replace the `...` in the next statement with the caps data.
See documentation
        // for `gst_caps_new_simple` for more information.

        GstCaps* caps = gst_caps_new_simple("cust/om", ...);
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, caps);
        gst_caps_unref(caps);
    }
}



On Mon, Jul 23, 2018 at 9:36 AM, David Ing <ding at panopto.com> wrote:

> I do not have an answer to your questions, but I noticed a documentation
> page:
>
> https://gstreamer.freedesktop.org/documentation/plugin-
> development/advanced/media-types.html#table-of-other-types
>
>
> At the very bottom of that page, I noticed an empty table "Table of Other
> Types" which seems to suggest that you can define any type you want (with
> any prefix, not just "audio/" or "video/").
>
> I am guessing that you would simply write a plugin with a typefind
> function like this ...
>
>
> // Executes typefind logic for "cust/om" media type
> // @param ptr  A pointer to the first byte of the stream.
> static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
> {
>     int64_t offset = 0;
>     uint32_t size = 100;
>     const guint8 *data = gst_type_find_peek(typeFind, offset, size);
>
>     if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
>     {
>         GstCaps* caps = ...
>         gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, gst_caps_new_simple("cust/om", caps));
>         gst_caps_unref(caps);
>     }
> }
>
> And in your `plugin_init` method, you would need to register the typefind
> function using `gst_type_find_register`.
>
> Notice that in the typefind function, you can define the caps of the
> stream (such as the dimensionality of your tensor).  I believe that the
> caps structure can contain any fields you want.
>
> Next your plugin should define a pad template having SINK caps which are
> compatible with the caps that would be detected by your typefind function
> .  Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ...
> but it needs to contain the code which transforms the stream from SINK to
> SRC.
>
> I can't see any reason why that wouldn't work, but I am just a novice:
> Someone should correct me if I am wrong.
>
> On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <myungjoo.ham at samsung.com>
> wrote:
>
>> Dear Gstreamer Developers,
>>
>>
>> We have been working on developing Gstreamer plugins to handle
>> general neural network models implemented with on-the-shelf frameworks
>> (such as Tensorflow or Caffe) as media filters.
>>
>>
>> I have a few questions on how to register a new custom "typefind" for
>> a new non-media type, "tensor/tensors".
>>
>>
>>
>> Because the "sources" are general media streams, such as "camera", "mic",
>> or "media files", the to-be-proposed type, "other/tensor" and
>> "other/tensors", do not have their file format, yet.
>> Their meta data are handled as pad-capabilities and
>> the buffer between elements contains the tensor data only
>> (similar to what video/x-raw does)
>>
>> However, assuming that we need to register "typefind",
>> later to be considered to become "good plugin" some day,
>> it appears that we need to define headers as if they are
>> stored as files. (for debugging and testing purpose, we may need this
>> anyway)
>>
>>
>>
>> Q1. Is it ok to define "other/tensor(s)" type to have some headers
>> (including dimensions of tensors) when it is stored in file and
>> not to have any meta data in inter-gst-element transfer (like
>> video/x-raw)?
>>
>> Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
>> the saved file format for tensors and use "other/tensor(s)" for
>> video/x-raw-like handling?
>>
>> Q3. Or is it ok not to register any and possible to upstream to
>> gst-plugins-good with these new media types (other/tensor and
>> other/tensors)?
>>
>>
>>
>> As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
>> this project for a while and returned), we've finally got internal
>> approval
>> to fully open sources (In LGPL-2.1.)
>> We will be migrating the code to github.com/nnsuite soon, probably in
>> this week.
>>
>>
>> The full migration requires CI/CD systems to be migrated as well,
>> so the migration of developing process might take some time.
>>
>>
>>
>> We hope, someday, we can upstream the plugins to gst-plugins-good. :)
>>
>>
>>
>>
>>
>> Cheers,
>> MyungJoo
>>
>> --
>>
>> MyungJoo Ham (함명주), Ph.D.
>> Autonomous Machine Lab., AI Center, Samsung Research.
>> Cell: +82-10-6714-2858
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180723/a1f90429/attachment.html>