[Spice-devel] [PATCH spice-server v2] gstreamer: Check if ORC library can work

[Frediano Ziglio]

> 
> On 08/18/2017 02:32 PM, Frediano Ziglio wrote:
> > ORC lirabry is used internally by GStreamer to generate code
> > dynamically.
> > If ORC cannot allocate executable memory the failure cause
> > an abort(3) to be called.
> > This happens on some SELinux configuration that disable executable
> > memory allocation (execmem boolean).
> > Check that ORC could work before attempting to use GStreamer to
> > avoid crashes. The log will report an error.
> > 
> > Signed-off-by: Frediano Ziglio <fziglio at redhat.com>
> > ---
> >   configure.ac               |  6 ++++++
> >   server/Makefile.am         |  2 ++
> >   server/gstreamer-encoder.c | 30 ++++++++++++++++++++++++++++++
> >   3 files changed, 38 insertions(+)
> > 
> 
> <snip>
> > +/* Check if ORC library can work.
> > + * ORC library is used quite extensively by GStreamer
> > + * to generate code dynamically. If ORC cannot work GStreamer
> > + * will abort(3) the entire process.
> > + */
> > +static bool orc_check(void)
> > +{
> > +    static bool orc_checked = false;
> > +    static bool orc_dynamic_code_ok = false;
> > +
> > +    if (SPICE_UNLIKELY(!orc_checked)) {
> > +        OrcCode *code = orc_code_new();
> > +        if (code) {
> > +            /* allocating 0 byte for code make the function not crash
> > +             * but doing all initializations and checks */
> 
> Hi Freidano,
> 
> man malloc suggests that if size is 0, NULL may be returned even
> if successful.
> 

orc_code_allocate_codemem has nothing to do with malloc,
the purpose is to allocate a writeble and executable
memory buffer which is forbidden by some SELinux settings

> Does it crash if allocating, but not writing/executing, e.g. 16 bytes  ?
> 

The function crash if called with a size != 0 in this case.

If is not clear could you suggest a better comment?
Or extend it.

> Uri.
> 
> > +            orc_code_allocate_codemem(code, 0);
> > +            orc_dynamic_code_ok = code->code != NULL;
> > +            orc_code_free(code);
> > +        }
> > +        orc_checked = true;
> > +    }
> > +    return orc_dynamic_code_ok;
> > +}
> > +
> >   VideoEncoder *gstreamer_encoder_new(SpiceVideoCodecType codec_type,
> >                                       uint64_t starting_bit_rate,
> >                                       VideoEncoderRateControlCbs *cbs,
> > @@ -1721,6 +1746,11 @@ VideoEncoder
> > *gstreamer_encoder_new(SpiceVideoCodecType codec_type,
> >           return NULL;
> >       }
> >   
> > +    // avoid aborting the process
> > +    if (!orc_check()) {
> > +        return NULL;
> > +    }
> > +
> >       SpiceGstEncoder *encoder = spice_new0(SpiceGstEncoder, 1);
> >       encoder->base.destroy = spice_gst_encoder_destroy;
> >       encoder->base.encode_frame = spice_gst_encoder_encode_frame;
> > 
> 
> 

Frediano