Why is GstBuffer not writable in the _fill method of GstPushSrc?

Ben Rush ben at ben-rush.net
Mon Apr 29 15:46:47 UTC 2019


Sebastian,

So, I'm not sure precisely what's going on, but I think it might - maybe -
have something to do with the fact that I'm using the MFX (Intel Media SDK)
GStreamer modules, and that perhaps things aren't playing well together.
Let me explain. I've got an example application that, all it does, is
encode a static image of the scientist Richard Feynman to disk as an MP4
file (500 frames). I use it to kind of tinker and explore the various parts
of the GStreamer pipeline.

I have found that if I set the output caps to NV12, the gst_buffer_map()
call fails:

static GstStaticPadTemplate gst_feynman_template =
GST_STATIC_PAD_TEMPLATE("src",
GST_PAD_SRC,
GST_PAD_ALWAYS,
GST_STATIC_CAPS(GST_VIDEO_CAPS_MAKE("{NV12}"))
);

If, on the other hand, I change this to AYUV, it works:

static GstStaticPadTemplate gst_feynman_template =
GST_STATIC_PAD_TEMPLATE("src",
GST_PAD_SRC,
GST_PAD_ALWAYS,
GST_STATIC_CAPS(GST_VIDEO_CAPS_MAKE("{AYUV}"))
);

It doesn't matter what I actually send as this failure happens before I
send anything at all. During the first invocation to fill the buffer, the
gst_buffer_map call fails as I've indicated in previous emails if I specify
NV12. As I'm writing this email out certain things are starting to becoming
clearer, and I presume what's happening is that per the handshake between
my source DLL and the downstream gstmfx.dll (and the x264 encoder therein)
something is failing to initialize properly, thereby causing the GStreamer
pipeline to fail to also initialize the GstBuffer for me properly.
Specifically, the gstmfxenc_h264.c encoder is failing to handle NV12
properly (or me asking it to handle this format is failing on its side and
therefore falling back to some standard GStreamer allocator/handler (I
don't know the terminology well as I'm still learning)).

I know that the gstmfx encoder suite advertises that it supports these
formats:

# define GST_MFX_SUPPORTED_INPUT_FORMATS \
    "{ NV12, YV12, I420, YUY2, P010_10LE, BGRA, BGRx }"

What tipped me off was inspecting the GstBuffer object in-memory, and
seeing when I specified NV12, the GstBuffer->Pool->Object->name was
"mfxvideobufferpool0", whereas if I specify AYUV, the name is
"videobufferpool0".

So, does this appear to be a bug in the intel media encoder side of things?
Or am I still not using something correctly? If it is a bug in the Intel
side of things, do you have any advise on how to track it down? They've
been pretty unhelpful so far regarding various other issues I've
encountered using their stuff.

Here is the full source to my Feynman encoder:

#include <gst/gst.h>
#include <gst/base/gstpushsrc.h>
#include <gst/video/gstvideometa.h>
#include <gst/video/gstvideopool.h>

#include <Windows.h>
#include <stdio.h>

#define PACKAGE "gst_feynman"

//
https://gstreamer.freedesktop.org/documentation/plugin-development/basics/boiler.html
/* Definition of structure storing data for this element. */
typedef struct _GstFeynman
{
GstPushSrc element;
GstPad* sourcepad;

/* running time and frames for current caps */
GstClockTime running_time;            /* total running time */
gint64 n_frames;                      /* total frames sent */
gboolean reverse;

/* previous caps running time and frames */
GstClockTime accum_rtime;              /* accumulated running_time */
gint64 accum_frames;                  /* accumulated frames */

GstVideoInfo info; /* protected by the object or stream lock */

  /* private */
  /* FIXME 2.0: Change type to GstClockTime */
gint64 timestamp_offset;              /* base offset */

gpointer* lines;

guint n_lines;
gint offset;
} GstFeynman;

/* Standard definition defining a class for this element. */
typedef struct _GstFeynmanClass
{
//https://github.com/GStreamer/gst-plugins-base/tree/master/gst/videotestsrc
//
https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrc.h
GstPushSrcClass parent_class;
} GstFeynmanClass;


/* Standard macros for defining types for this element.  */
#define GST_FEYNMAN_TYPE (gst_feynman_get_type())
#define GST_FEYNMAN(obj) \
  (G_TYPE_CHECK_INSTANCE_CAST((obj),GST_FEYNMAN_TYPE,GstFeynman))
#define GST_FEYNMAN_CLASS(klass) \
  (G_TYPE_CHECK_CLASS_CAST((klass),GST_FEYNMAN_TYPE,GstFeynmanClass))
#define GST_IS_FEYNMAN(obj) \
  (G_TYPE_CHECK_INSTANCE_TYPE((obj),GST_FEYNMAN_TYPE))
#define GST_IS_FEYNMAN_CLASS(klass) \
  (G_TYPE_CHECK_CLASS_TYPE((klass),GST_FEYNMAN_TYPE))

/* Standard function returning type information. */
GType gst_feynman_get_type(void);

#define gst_feynman_parent_class parent_class
G_DEFINE_TYPE(GstFeynman, gst_feynman, GST_TYPE_PUSH_SRC);

#define VTS_VIDEO_CAPS GST_VIDEO_CAPS_MAKE (GST_VIDEO_FORMATS_ALL)


static GstStaticPadTemplate gst_feynman_template =
GST_STATIC_PAD_TEMPLATE("src",
GST_PAD_SRC,
GST_PAD_ALWAYS,
GST_STATIC_CAPS(GST_VIDEO_CAPS_MAKE("{AYUV}"))
);

static void gst_feynman_set_property(GObject* object, guint prop_id,
const GValue* value, GParamSpec* pspec);
static void gst_feynman_get_property(GObject* object, guint prop_id,
GValue* value, GParamSpec* pspec);
static GstFlowReturn
gst_feynman_src_fill(GstPushSrc* psrc, GstBuffer* buffer);
static gboolean gst_feynman_start(GstBaseSrc* basesrc);
static gboolean gst_feynman_stop(GstBaseSrc* basesrc);
static gboolean gst_feynman_set_caps(GstBaseSrc* bsrc, GstCaps* caps);
static gboolean gst_feynman_is_seekable(GstBaseSrc* src);
static GstCaps* gst_feynman_src_fixate(GstBaseSrc* bsrc, GstCaps* caps);

/* initialize the myfilter's class */
static void gst_feynman_class_init(GstFeynmanClass* klass)
{
// get the "object class".
GObjectClass* gobject_class = (GObjectClass*)klass;
GstElementClass* gstelement_class = (GstElementClass*)klass;
GstBaseSrcClass* gstbasesrc_class = (GstBaseSrcClass*)klass;
GstPushSrcClass* gstpushsrc_class = (GstPushSrcClass*)klass;

// set the getters and setters for properties.
gobject_class->set_property = gst_feynman_set_property;
gobject_class->get_property = gst_feynman_get_property;

gst_element_class_set_static_metadata(gstelement_class,
"Video test source", "Source/Video",
"Creates a test video stream", "David A. Schleef <ds at schleef.org>");

// assign the pad to this class.
gst_element_class_add_static_pad_template(gstelement_class,
&gst_feynman_template);

gstbasesrc_class->fixate = gst_feynman_src_fixate;
gstbasesrc_class->is_seekable = gst_feynman_is_seekable;
gstbasesrc_class->set_caps = gst_feynman_set_caps;
gstbasesrc_class->start = gst_feynman_start;
gstbasesrc_class->stop = gst_feynman_stop;
gstpushsrc_class->fill = gst_feynman_src_fill;
}

static GstCaps* gst_feynman_src_fixate(GstBaseSrc* bsrc, GstCaps* caps)
{
GstFeynman* src = GST_FEYNMAN(bsrc);
GstStructure* structure;

caps = gst_caps_make_writable(caps);
structure = gst_caps_get_structure(caps, 0);

gst_structure_fixate_field_nearest_int(structure, "width", 320);
gst_structure_fixate_field_nearest_int(structure, "height", 240);

if (gst_structure_has_field(structure, "framerate"))
gst_structure_fixate_field_nearest_fraction(structure, "framerate", 30, 1);
else
gst_structure_set(structure, "framerate", GST_TYPE_FRACTION, 30, 1, NULL);

if (gst_structure_has_field(structure, "pixel-aspect-ratio"))
gst_structure_fixate_field_nearest_fraction(structure,
"pixel-aspect-ratio", 1, 1);
else
gst_structure_set(structure, "pixel-aspect-ratio", GST_TYPE_FRACTION, 1, 1,
NULL);

if (gst_structure_has_field(structure, "colorimetry"))
gst_structure_fixate_field_string(structure, "colorimetry", "bt601");
if (gst_structure_has_field(structure, "chroma-site"))
gst_structure_fixate_field_string(structure, "chroma-site", "mpeg2");

if (gst_structure_has_field(structure, "interlace-mode"))
gst_structure_fixate_field_string(structure, "interlace-mode",
"progressive");
else
gst_structure_set(structure, "interlace-mode", G_TYPE_STRING,
"progressive", NULL);

if (gst_structure_has_field(structure, "multiview-mode"))
gst_structure_fixate_field_string(structure, "multiview-mode",
gst_video_multiview_mode_to_caps_string
(GST_VIDEO_MULTIVIEW_MODE_MONO));
else
gst_structure_set(structure, "multiview-mode", G_TYPE_STRING,
gst_video_multiview_mode_to_caps_string(GST_VIDEO_MULTIVIEW_MODE_MONO),
NULL);

caps = GST_BASE_SRC_CLASS(parent_class)->fixate(bsrc, caps);

return caps;
}


static gboolean gst_feynman_is_seekable(GstBaseSrc* psrc)
{
/* we're seekable... */
return FALSE;
}

static gboolean gst_feynman_set_caps(GstBaseSrc* bsrc, GstCaps* caps)
{
const GstStructure* structure;
GstFeynman* feynman;
GstVideoInfo info;
guint i;
guint n_lines = 1;
gint offset = 0;

feynman = GST_FEYNMAN(bsrc);

structure = gst_caps_get_structure(caps, 0);

GST_OBJECT_LOCK(feynman);

gst_video_info_from_caps(&info, caps);

//info.width = 320;
//info.height = 240;
//info.fps_n = 30;
//info.fps_d = 1;

feynman->lines = (gpointer*)g_malloc(sizeof(gpointer) * n_lines);
for (i = 0; i < n_lines; i++)
feynman->lines[i] = g_malloc((info.width + 16) * 8);
feynman->n_lines = n_lines;
feynman->offset = offset;

/* looks ok here */
feynman->info = info;

GST_OBJECT_UNLOCK(feynman);

return TRUE;
}

static gboolean gst_feynman_stop(GstBaseSrc * basesrc)
{
return TRUE;
}

static gboolean gst_feynman_start(GstBaseSrc * basesrc)
{
GstFeynman* src = GST_FEYNMAN(basesrc);

GST_OBJECT_LOCK(src);
src->running_time = 0;
src->n_frames = 0;
src->accum_frames = 0;
src->accum_rtime = 0;

gst_video_info_init(&src->info);
GST_OBJECT_UNLOCK(src);

return TRUE;
}

static void gst_feynman_init(GstFeynman * src)
{
OutputDebugStringA("Feynman: gst_feynman_init called.");

gst_base_src_set_format(GST_BASE_SRC(src), GST_FORMAT_TIME);
gst_base_src_set_live(GST_BASE_SRC(src), 1);

return;
}

static gboolean plugin_init(GstPlugin * plugin)
{
OutputDebugStringA("Feynman: plugin_init called.");

return gst_element_register(plugin, "gstfeynman",
GST_RANK_NONE, GST_FEYNMAN_TYPE);
}

static gboolean gst_feynman_query(GstBaseSrc * bsrc, GstQuery * query)
{
return GST_BASE_SRC_CLASS(parent_class)->query(bsrc, query);
}

static void gst_feynman_set_property(GObject * object, guint prop_id,
const GValue * value, GParamSpec * pspec)
{
return;
}

static void gst_feynman_get_property(GObject * object, guint prop_id,
GValue * value, GParamSpec * pspec)
{
return;
}


#define CLIP(X) ( (X) > 255 ? 255 : (X) < 0 ? 0 : X)

// RGB -> YUV
#define RGB2Y(R, G, B) CLIP(( (  66 * (R) + 129 * (G) +  25 * (B) + 128) >>
8) +  16)
#define RGB2U(R, G, B) CLIP(( ( -38 * (R) -  74 * (G) + 112 * (B) + 128) >>
8) + 128)
#define RGB2V(R, G, B) CLIP(( ( 112 * (R) -  94 * (G) -  18 * (B) + 128) >>
8) + 128)

static GstFlowReturn gst_feynman_src_fill(GstPushSrc * psrc,
GstBuffer * buffer)
{
GstFeynman* src;
GstClockTime next_time;
GstVideoFrame frame;
gconstpointer pal;
gsize palsize;

src = GST_FEYNMAN(psrc);

printf(".");

if (G_UNLIKELY(GST_VIDEO_INFO_FORMAT(&src->info) ==
GST_VIDEO_FORMAT_UNKNOWN))
goto not_negotiated;

/* 0 framerate and we are at the second frame, eos */
if (G_UNLIKELY(src->info.fps_n == 0 && src->n_frames == 1))
goto eos;

if (G_UNLIKELY(src->n_frames == -1)) {
/* EOS for reverse playback */
goto eos;
}

GST_LOG_OBJECT(src,
"creating buffer from pool for frame %" G_GINT64_FORMAT, src->n_frames);

GST_BUFFER_PTS(buffer) =
src->accum_rtime + src->timestamp_offset + src->running_time;
GST_BUFFER_DTS(buffer) = GST_CLOCK_TIME_NONE;

gst_object_sync_values(GST_OBJECT(psrc), GST_BUFFER_PTS(buffer));

///////////////////////// SHOW FEYNMAN 1////////////////////////////////////
//if (!gst_video_frame_map(&frame, &src->info, buffer, GST_MAP_WRITE))
// goto invalid_frame;

//FILE* f = fopen("feynman.dat", "rb+");
//int* line = (int*)malloc(frame.info.width * 4);
//unsigned char lineBuffer[320 * 3];
//for (int j = 0; j < frame.info.height; j++)
//{
// fread(lineBuffer, 1, sizeof(lineBuffer), f);

// for (int i = 0; i < frame.info.width; i++)
// {
// int b = lineBuffer[i * 3];
// int g = lineBuffer[i * 3 + 1];
// int r = lineBuffer[i * 3 + 2];

// int Y = r * .299000 + g * .587000 + b * .114000;
// int U = r * -.168736 + g * -.331264 + b * .500000 + 128;
// int V = r * .500000 + g * -.418688 + b * -.081312 + 128;

// int value = (0 << 0) | (Y << 8) | (U << 16) | ((guint32) V << 24);

// line[i] = value;
// }

// frame.info.finfo->pack_func(frame.info.finfo,
// GST_VIDEO_PACK_FLAG_NONE, line, 0, frame.data, frame.info.stride,
// frame.info.chroma_site, j, frame.info.width);
//}
//free(line);
//fclose(f);

//if ((pal = gst_video_format_get_palette(GST_VIDEO_FRAME_FORMAT(&frame),
// &palsize))) {
// memcpy(GST_VIDEO_FRAME_PLANE_DATA(&frame, 1), pal, palsize);
//}

//gst_video_frame_unmap(&frame);
/////////////////////////////////////////////////////////////////////////

////////////////////////////SHOW FEYNMAN 2//////////////////////////////
FILE* f = fopen("feynman.yuv", "rb+");
fseek(f, 0, SEEK_END);
int length = ftell(f);
fseek(f, 0, SEEK_SET);
GstMapInfo info;
gst_buffer_map(buffer, &info, GST_MAP_WRITE);
char* buff = (char*)malloc(length);
fread(buff, 1, length, f);
memcpy(info.data, buff, length);
gst_buffer_unmap(buffer, &info);
fclose(f);
///////////////////////////////////////////////////////////////////////


// the data becomes planar when it is sent downstream.
//GstMapInfo info;
//gst_buffer_map(buffer, &info, GST_MAP_READ);
//FILE* fout = fopen("feynman.yuv", "wb+");
//fwrite(info.data, 1, info.size, fout);
//fclose(fout);
//gst_buffer_unmap(buffer, &info);

GST_DEBUG_OBJECT(src, "Timestamp: %" GST_TIME_FORMAT " = accumulated %"
GST_TIME_FORMAT " + offset: %"
GST_TIME_FORMAT " + running time: %" GST_TIME_FORMAT,
GST_TIME_ARGS(GST_BUFFER_PTS(buffer)), GST_TIME_ARGS(src->accum_rtime),
GST_TIME_ARGS(src->timestamp_offset), GST_TIME_ARGS(src->running_time));

GST_BUFFER_OFFSET(buffer) = src->accum_frames + src->n_frames;
if (src->reverse) {
src->n_frames--;
}
else {
src->n_frames++;
}
GST_BUFFER_OFFSET_END(buffer) = GST_BUFFER_OFFSET(buffer) + 1;
if (src->info.fps_n) {
next_time = gst_util_uint64_scale(src->n_frames,
src->info.fps_d * GST_SECOND, src->info.fps_n);
if (src->reverse) {
GST_BUFFER_DURATION(buffer) = src->running_time - next_time;
}
else {
GST_BUFFER_DURATION(buffer) = next_time - src->running_time;
}
}
else {
next_time = src->timestamp_offset;
/* NONE means forever */
GST_BUFFER_DURATION(buffer) = GST_CLOCK_TIME_NONE;
}

src->running_time = next_time;

return GST_FLOW_OK;

not_negotiated:
{
return GST_FLOW_NOT_NEGOTIATED;
}
eos:
{
GST_DEBUG_OBJECT(src, "eos: 0 framerate, frame %d", (gint)src->n_frames);
return GST_FLOW_EOS;
}
invalid_frame:
{
GST_DEBUG_OBJECT(src, "invalid frame");
return GST_FLOW_OK;
}
}

GST_PLUGIN_DEFINE(
GST_VERSION_MAJOR,
GST_VERSION_MINOR,
feynman,
"feynman",
plugin_init,
".1",
"LGPL",
"gst_feynman",
"http://gstreamer.net/"
)


On Mon, Apr 29, 2019 at 5:31 AM Ben Rush <ben at ben-rush.net> wrote:

> Yes, and I believe I have possibly narrowed it down. I should be able to
> provide a self-contained example as well as even a cause for it sometime
> today.
>
> Thank you for your response.
>
> On Mon, Apr 29, 2019, 01:29 Sebastian Dröge <sebastian at centricular.com
> wrote:
>
>> On Sun, 2019-04-28 at 10:23 -0500, Ben Rush wrote:
>> > Just to be clear, I'm seeing examples of others doing what I want to
>> > do. That is, write directly to the GstBuffer as a GstPushSrc.
>> > Examples include:
>> >
>> >
>> https://github.com/GStreamer/gst-plugins-bad/blob/master/gst/frei0r/gstfrei0rsrc.c
>> (between lines 96 to
>> > 106)
>> >
>> https://gitlab.collabora.com/gkiagia/gst-plugins-bad/blob/8cdfb13658a069cf8c45a3265bf865849d3dc8e9/ext/neon/gstneonhttpsrc.c
>> (between lines 955 to
>> > 995)
>> >
>> > Among others. So, I don't feel like what I'm doing is strange or out
>> > of the ordinary. It's just that the map function call fails (returns
>> > false) and gives me a null buffer to write to when I try to open it
>> > with GST_MAP_WRITE. It succeeds when I try to open it with
>> > GST_MAP_READ, but that makes no sense when I'm trying to WRITE to the
>> > buffer.
>>
>> Can you provide a standalone testcase for this so we have an idea what
>> your code is doing exactly and a way to reproduce it?
>>
>> As you say yourself, it's rather useless to have a non-writable buffer
>> inside fill() and that definitely shouldn't happen.
>>
>> --
>> Sebastian Dröge, Centricular Ltd · https://www.centricular.com
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20190429/769802d2/attachment-0001.html>


More information about the gstreamer-devel mailing list