<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> Hi,<o:p></o:p> <o:p> </o:p> Thanks one more time, again that was very helpful. I manage to retrieve ROI meta form video frames.<o:p></o:p> <o:p> </o:p> I have one more question GstAudioLevelMeta and voice_activity Boolean.<o:p></o:p> <o:p> </o:p> I have simple code with this pipeline<o:p></o:p> <o:p> </o:p> pipeline = Gst.parse_launch("directsoundsrc ! \<o:p></o:p> level audio-level-meta=true post-messages=false message=false name=elLev ! \<o:p></o:p> webrtcdsp name=dsp echo-cancel=false gain-control=false high-pass-filter=false limiter=false noise-suppression=false voice-detection=true voice-detection-likelihood=high voice-detection-frame-size-ms=10 ! \<o:p></o:p> appsink name=sink emit-signals=true")<o:p></o:p> <o:p> </o:p> appsink = pipeline.get_by_name("sink")<o:p></o:p> appsink.connect("new-sample", on_buffer, None)<o:p></o:p> <o:p> </o:p> and <o:p></o:p> <o:p> </o:p> def on_buffer(sink: GstApp.AppSink, data: typ.Any) -> Gst.FlowReturn:<o:p></o:p> <o:p> </o:p> sample = sink.emit("pull-sample") # Gst.Sample<o:p></o:p> buffer = sample.get_buffer() # Gst.Buffer<o:p></o:p> ll = GstAudio.buffer_get_audio_level_meta(buffer)<o:p></o:p> text_file.write(str(ll.voice_activity) + '\n')<o:p></o:p> <o:p> </o:p> return Gst.FlowReturn.OK<o:p></o:p> <o:p> </o:p> If I use bus message from dsp element to check for voice activity I get correct results more or less. When Im speaking stream-has-voice is on true and when I stop its changes to false.<o:p></o:p> <o:p> </o:p> But if I use audio_level_meta I never get 2 consecutive buffers with voice_activity set to True.<o:p></o:p> If I start speaking I get one buffer with voice_activity set to True and next one is False and it stays False until I stop speaking and start speaking again then again I get one buffer with True and next one is again False.<o:p></o:p> <o:p> </o:p> Do I miss something here? Or this is expected behavior.<o:p></o:p> <o:p> </o:p> Br,<o:p></o:p> Dejan<o:p></o:p> <o:p> </o:p> From: Nicolas Dufresne <nicolas@ndufresne.ca> Sent: Friday, April 15, 2022 7:54 PM To: Dejan Cotra <Dejan.Cotra@nttdata.com> Cc: Discussion of the development of and with GStreamer <gstreamer-devel@lists.freedesktop.org> Subject: Re: webrtcdsp voice detection<o:p></o:p> <o:p> </o:p> <div> <div> <o:p> </o:p> <div> <div> Le ven. 15 avr. 2022 08 h 47, Dejan Cotra <<a href="mailto:Dejan.Cotra@nttdata.com">Dejan.Cotra@nttdata.com</a>> a écrit :<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> Hi Nicolas, Thank you that was very helpful. I have one similar question. I also play around with facedetect element. I know that I can retrieve information about face from bus message emitted by facedetect element. Is there a way to retrieve informations about face from video frame metainfo? Something similar to voice_activity in GstAudioLevelMeta?<o:p></o:p> </blockquote> </div> </div> <div> <o:p> </o:p> </div> <div> I'm currently away from a real computer to check the code, though the "in-band" way is GstVideoRegionOfInterest. ROI are rectangles with a type (a simple string). So facedetect should be adding ROI meta to the frames. <o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> A typical use case is to detect these with a pad probe and set a qp-delta (not sure of the name) so that capable encoders can be told to work harder on the details of that rectangle.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> This could also be used for other purposes. Note that ONNX (ML plugins we have) tend to prefer having more shapes then just rectangles, so they have their own meta.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> <div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> Br, Dejan -----Original Message----- From: Nicolas Dufresne <<a href="mailto:nicolas@ndufresne.ca" target="_blank">nicolas@ndufresne.ca</a>> Sent: Friday, April 8, 2022 3:31 PM To: Discussion of the development of and with GStreamer <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> Cc: Dejan Cotra <<a href="mailto:Dejan.Cotra@nttdata.com" target="_blank">Dejan.Cotra@nttdata.com</a>> Subject: Re: webrtcdsp voice detection Le vendredi 08 avril 2022 à 11:10 +0000, Dejan Cotra via gstreamer-devel a écrit : [...] > > I know that I can retrieve informations from webrtcdsp voice detection > via bus messages. I receive GST_MESSAGE_ELEMENT message from webrtcdsp > element with payload like this: > > voice-activity, stream-time=(guint64)2640000000, stream-has- > voice=(boolean)false; > > Question is can I retrieve informations about voice detection in some > other way. Like metainfo of each sample that I pull from appsink > element? Or something similar? It also sets the voice_activity boolean in GstAudioLevelMeta (along with the audio amplitude). This is per buffers, not per samples. So you get feedback every 10ms more or less. Nicolas<o:p></o:p> </blockquote> </div> </div> </div> </div> </body> </html>