Adding typefind function for a new Gst Type, "tensors"

[David Ing]

I do not have an answer to your questions, but I noticed a documentation
page:

https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html#table-of-other-types


At the very bottom of that page, I noticed an empty table "Table of Other
Types" which seems to suggest that you can define any type you want (with
any prefix, not just "audio/" or "video/").

I am guessing that you would simply write a plugin with a typefind function
like this ...


// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' &&
data[3] == 'T')
    {
        GstCaps* caps = ...
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM,
gst_caps_new_simple("cust/om", caps));
        gst_caps_unref(caps);
    }
}

And in your `plugin_init` method, you would need to register the typefind
function using `gst_type_find_register`.

Notice that in the typefind function, you can define the caps of the
stream (such
as the dimensionality of your tensor).  I believe that the caps structure
can contain any fields you want.

Next your plugin should define a pad template having SINK caps which are
compatible with the caps that would be detected by your typefind function
.  Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ...
but it needs to contain the code which transforms the stream from SINK to
SRC.

I can't see any reason why that wouldn't work, but I am just a novice:
Someone should correct me if I am wrong.

On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <myungjoo.ham at samsung.com>
wrote:

> Dear Gstreamer Developers,
>
>
> We have been working on developing Gstreamer plugins to handle
> general neural network models implemented with on-the-shelf frameworks
> (such as Tensorflow or Caffe) as media filters.
>
>
> I have a few questions on how to register a new custom "typefind" for
> a new non-media type, "tensor/tensors".
>
>
>
> Because the "sources" are general media streams, such as "camera", "mic",
> or "media files", the to-be-proposed type, "other/tensor" and
> "other/tensors", do not have their file format, yet.
> Their meta data are handled as pad-capabilities and
> the buffer between elements contains the tensor data only
> (similar to what video/x-raw does)
>
> However, assuming that we need to register "typefind",
> later to be considered to become "good plugin" some day,
> it appears that we need to define headers as if they are
> stored as files. (for debugging and testing purpose, we may need this
> anyway)
>
>
>
> Q1. Is it ok to define "other/tensor(s)" type to have some headers
> (including dimensions of tensors) when it is stored in file and
> not to have any meta data in inter-gst-element transfer (like video/x-raw)?
>
> Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
> the saved file format for tensors and use "other/tensor(s)" for
> video/x-raw-like handling?
>
> Q3. Or is it ok not to register any and possible to upstream to
> gst-plugins-good with these new media types (other/tensor and
> other/tensors)?
>
>
>
> As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
> this project for a while and returned), we've finally got internal approval
> to fully open sources (In LGPL-2.1.)
> We will be migrating the code to github.com/nnsuite soon, probably in
> this week.
>
>
> The full migration requires CI/CD systems to be migrated as well,
> so the migration of developing process might take some time.
>
>
>
> We hope, someday, we can upstream the plugins to gst-plugins-good. :)
>
>
>
>
>
> Cheers,
> MyungJoo
>
> --
>
> MyungJoo Ham (함명주), Ph.D.
> Autonomous Machine Lab., AI Center, Samsung Research.
> Cell: +82-10-6714-2858
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180723/a564ed3c/attachment-0001.html>