[alsa-devel] Need your advice: Add a new communication inteface between HD-Audio and Gfx drivers for hotplug notification/ELD update

[Jaroslav Kysela]

Date 23.1.2014 08:57, Takashi Iwai wrote:
> At Thu, 23 Jan 2014 06:35:12 +0000,
> Lin, Mengdong wrote:
>>
>>> -----Original Message-----
>>> From: Takashi Iwai [mailto:tiwai at suse.de]
>>> Sent: Thursday, January 23, 2014 1:19 AM
>>> To: Daniel Vetter
>>> Cc: Lin, Mengdong; Barnes, Jesse; Zanoni, Paulo R;
>>> alsa-devel at alsa-project.org; intel-gfx at lists.freedesktop.org; dri-devel
>>> Subject: Re: Need your advice: Add a new communication inteface
>>> between HD-Audio and Gfx drivers for hotplug notification/ELD update
>>>
>>> At Wed, 22 Jan 2014 15:18:21 +0100,
>>> Daniel Vetter wrote:
>>>>
>>>> On Wed, Jan 22, 2014 at 12:48:04PM +0000, Lin, Mengdong wrote:
>>>>>> -----Original Message-----
>>>>>> From: daniel.vetter at ffwll.ch [mailto:daniel.vetter at ffwll.ch] On
>>>>>> Behalf Of Daniel Vetter
>>>>>> Sent: Tuesday, January 21, 2014 9:11 PM
>>>>>> To: Lin, Mengdong
>>>>>> Cc: Takashi Iwai (tiwai at suse.de); Barnes, Jesse; Zanoni, Paulo R;
>>>>>> alsa-devel at alsa-project.org; intel-gfx at lists.freedesktop.org
>>>>>> Subject: Re: Need your advice: Add a new communication inteface
>>>>>> between HD-Audio and Gfx drivers for hotplug notification/ELD
>>>>>> update
>>>>>>
>>>>>> On Tue, Jan 21, 2014 at 1:35 PM, Lin, Mengdong
>>>>>> <mengdong.lin at intel.com>
>>>>>> wrote:
>>>>>>> Dear audio and gfx stakeholders,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We hope to add a new interface between audio and gfx driver, for
>>>>>>> gfx driver to notify audio about HDMI/DP hot-plug and ELD update.
>>>>>>>
>>>>>>> Would you please share some comments on the proposal below?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Background of this issue: On Intel Haswell/Broadwell platforms,
>>>>>>> there is a HW restriction that after the display HD-Audio
>>>>>>> controller is in D3,
>>>>>>>
>>>>>>> it cannot be waken up by HDMI/DP hot-plug. Consequently,
>>>>>>> although the gfx driver can still detect the HDMI/DP hot-plug,
>>>>>>>
>>>>>>> audio driver has no idea about this and cannot notify user space
>>>>>>> whether the external HDMI/DP monitor is available for audio
>>>>>>> playback,
>>>>>>>
>>>>>>> because the audio controller cannot wake up to D0 and receive HW
>>>>>>> unsolicited event about hot-plug from the audio codec.
>>>>>>>
>>>>>>> This limitation will affect user space to decide whether we can
>>>>>>> output audio over HDMI/DP.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> To solve the above limitation, Takashi suggested to add a new
>>>>>>> communication interface between audio and gfx driver: create a
>>>>>> common
>>>>>>> object
>>>>>>>
>>>>>>> containing the ops registered by both graphics and audio
>>>>>>> drivers, then communicate through it, something like
>>> vga_switcheroo.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is it okay to create this kernel object in i915 driver?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I915 can export an API like "display_register_audio_client" for
>>>>>>> audio driver to register a client and hot-plug notification ops.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I915 can also call some API like "display_register_gfx_client"
>>>>>>> itself and register ops for audio driver to query monitor
>>>>>>> presence and ELD info on a specific port.
>>>>>>>
>>>>>>> It would be faster for audio driver than quering ELD by
>>>>>>> command/response over the HD-A bus, thus avoid delay in i915
>>>>>>> mode
>>>>>> set.
>>>>>>>
>>>>>>> This will also avoid waking up the audio devices unnecessarily
>>>>>>> if the user space does not really want to use HDMI/DP for audio
>>> playback.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Whenever i195 enables/disables audio on a port in modeset, it
>>>>>>> can call some API like "display_set_audio_state()" on this
>>>>>>> kernel object and trigger notifications to the audio driver.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> When the audio driver is probed (in the delayed probe stage), it
>>>>>>> can request
>>>>>>> i915 API symbol to register the audio client for this
>>>>>>> communication kernel object.
>>>>>>>
>>>>>>> Since the 1st i915 mode set may happen before audio driver
>>>>>>> registers the ops, we'll let audio driver check ELD once after
>>>>>>> registering the audio client ops.
>>>>>>>
>>>>>>> And for the platforms which uses this communication interface,
>>>>>>> we can disable unsolicited event for HDMI/DP hot-plug in the audio
>>> driver.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We hope to hear your feedback and start to work out more details.
>>>>>
>>>>> Thanks for your advice, Daniel!
>>>>>
>>>>>> Yeah, I've discussed this at KS with Takashi and we've agreed that
>>>>>> some common object to facilitate driver interactions. A few things
>>>>>> though:
>>>>>> - This should be common infrastructure useable by all alsa and drm
>>>>>> drivers, not just i915 and snd-hda. Especially on embedded
>>>>>> platforms this issue is fairly rampant ...
>>>>>
>>>>> Agree. Where to put this common object?
>>>>> Is it okay to put it under /driver/gpu/drm, similar to vga_switchroo?
>>>>> Shall we divide clients into audio and gfx categories, and define
>>>>> different ops for them? Since different info/request flow in
>>>>> different direction between audio and gfx.
>>>>
>>>> I guess we could place them into drivers/gpu, yeah. For a name I'd
>>>> suggest avsink or something like that, to make it clear that it's the
>>>> combination of audio+video. For the actual interfaces I guess we just
>>>> need one object in the device model, but the interface should be split
>>>> into things called from the audio side only, functions for the video
>>>> driver side only and stuff which can be called from both sides. This
>>>> matters mostly just so we don't end up with deadlocks since we need a
>>>> lock to protect the avsink state itself (e.g. the EDL or the
>>> audio_output_connected state).
>>>>
>>>>>> - While at it it should also encompass power management handling
>>>>>> of the shared hw imo so that we can get rid of the hsw specific
>>>>>> hacks for the power well code. Or at least we need to rework the
>>>>>> power well code to reuse this new infrastructure, I don't really
>>>>>> want to maintain a few copies of the lazy symbol_get logic this kind
>>> of stuff requires.
>>>>>
>>>>> Sounds good.
>>>>>
>>>>>> - I think the biggest problem is figuring out who should register
>>>>>> these device nodes. I think it makes the most sense if we do this
>>>>>> in the gfx driver, but that requires some trickery on the alsa
>>>>>> side (probably with using -EPROBE_DEFER or something like that.
>>>>>
>>>>> Can the new infrastructure allow audio driver to query whether gfx
>>> driver is ready?
>>>>> Maybe audio can wait until gfx is ready. For HD-Audio driver, the
>>>>> most time consuming part is delayed after the probe stage, and
>>>>> actually we can wait in the delayed phase.
>>>>
>>>> Tbh I haven't really thought about this really. EPROBE_DEFER looks
>>>> like the technique used by embedded platforms, but there's also the
>>>> new aggregate device driver infrastructure that Russell King is
>>>> working on for the imx driver. Or maybe we need to hand-roll our own
>>> notification scheme.
>>>>
>>>> On a hunch it's probably best if the gfx side registers this device
>>>> (since it also owns the output state in general) and that the audio
>>>> side waits until the gfx side has registered everything if it's not
>>>> there yet. I also haven't though about how the audio side could probe
>>>> for the right avsink node really ...
>>>
>>> Yes, I think doing it first in the gfx side makes sense, too.
>>>
>>>>>> - I agree that passing ELD and all the other information through
>>>>>> this new structure makes lot more sense than the current mess we
>>>>>> have with passing the ELD through some hardware buffer.
>>>>>
>>>>>> - Finally I think we should assign some identifier to this link
>>>>>> which will get exposed both on the drm side and in alsa, so that
>>>>>> userspace can figure out which display connects to which output.
>>>>>> With that media player could do the Right Thing and automatically
>>>>>> place the audio stream on the right pin in alsa.
>>>>>
>>>>> Is there something that blocks media player from doing the right thing
>>> now?
>>>>> For HD-Audio, the eld entries under /proc/asound/cardx expose the
>>>>> ELD info and can help user space to check if a monitor is usable on a
>>> pin.
>>>>> Current limitation is these eld entries cannot update if the audio
>>>>> controller is in D3, so we need the new infrastructure to notify
>>>>> audio driver to update them. But I'm not sure about embedded audio,
>>>>> maybe Takashi would like to share more info.
>>>>
>>>> ELD doesn't contain the serial number from the EDID, so if you have
>>>> two monitors of the same model userspace can't figure out which audio
>>>> output is connected to which screen.
>>>
>>> Right.  And, comparing two different data just for knowing whether A
>>> and B are relevant is merely a pain.
>>
>> Thanks for clarification!
>> Maybe we can add output info (eg. display port number) to the eld entries under /proc/asound/cardx. Is it okay?
> 
> It's possible, but the proc file is just a help.  It can't be the
> API.  For accessing the information, we'll need some new API, or let
> inform via sysfs of the new device.

I agree here. From my view, the cleanest solution is to create an
universal API in the kernel for the shared state / binding information
and inter-communication. I believe, that other drivers may use this API
in the future like mentioned vga_switcheroo.

						Jaroslav

> 
>> And I have a question: how to assure the audio/gfx client find its right peer?
>> On a x86 platform, there can be an integrated GPU and an discrete GPU. So there can be two audio controllers and two GPUs.
>> We need to assure audio controller find the proper GPU, and vice versa. Maybe we need the peer audio/gfx to register with a same identifier (something like vendor ID) for peering.
> 
> Yes, it's an open issue.  So far, the binding with vga_switcheroo is
> done by a rough guess with PCI ID.
> 
> 
> Takashi
> _______________________________________________
> Alsa-devel mailing list
> Alsa-devel at alsa-project.org
> http://mailman.alsa-project.org/mailman/listinfo/alsa-devel
> 


-- 
Jaroslav Kysela <perex at perex.cz>
Linux Kernel Sound Maintainer
ALSA Project; Red Hat, Inc.