[Spice-devel] [PATCH spice-server v2] gstreamer: Check if ORC library can work

[Uri Lublin]

On 08/18/2017 02:32 PM, Frediano Ziglio wrote:
> ORC lirabry is used internally by GStreamer to generate code
> dynamically.
> If ORC cannot allocate executable memory the failure cause
> an abort(3) to be called.
> This happens on some SELinux configuration that disable executable
> memory allocation (execmem boolean).
> Check that ORC could work before attempting to use GStreamer to
> avoid crashes. The log will report an error.
> 
> Signed-off-by: Frediano Ziglio <fziglio at redhat.com>
> ---
>   configure.ac               |  6 ++++++
>   server/Makefile.am         |  2 ++
>   server/gstreamer-encoder.c | 30 ++++++++++++++++++++++++++++++
>   3 files changed, 38 insertions(+)
> 

<snip>
> +/* Check if ORC library can work.
> + * ORC library is used quite extensively by GStreamer
> + * to generate code dynamically. If ORC cannot work GStreamer
> + * will abort(3) the entire process.
> + */
> +static bool orc_check(void)
> +{
> +    static bool orc_checked = false;
> +    static bool orc_dynamic_code_ok = false;
> +
> +    if (SPICE_UNLIKELY(!orc_checked)) {
> +        OrcCode *code = orc_code_new();
> +        if (code) {
> +            /* allocating 0 byte for code make the function not crash
> +             * but doing all initializations and checks */

Hi Freidano,

man malloc suggests that if size is 0, NULL may be returned even
if successful.

Does it crash if allocating, but not writing/executing, e.g. 16 bytes  ?

Uri.

> +            orc_code_allocate_codemem(code, 0);
> +            orc_dynamic_code_ok = code->code != NULL;
> +            orc_code_free(code);
> +        }
> +        orc_checked = true;
> +    }
> +    return orc_dynamic_code_ok;
> +}
> +
>   VideoEncoder *gstreamer_encoder_new(SpiceVideoCodecType codec_type,
>                                       uint64_t starting_bit_rate,
>                                       VideoEncoderRateControlCbs *cbs,
> @@ -1721,6 +1746,11 @@ VideoEncoder *gstreamer_encoder_new(SpiceVideoCodecType codec_type,
>           return NULL;
>       }
>   
> +    // avoid aborting the process
> +    if (!orc_check()) {
> +        return NULL;
> +    }
> +
>       SpiceGstEncoder *encoder = spice_new0(SpiceGstEncoder, 1);
>       encoder->base.destroy = spice_gst_encoder_destroy;
>       encoder->base.encode_frame = spice_gst_encoder_encode_frame;
>