VGA arbitration: API proposal

[Benjamin Herrenschmidt]

On Sat, 2005-03-05 at 21:50 +0100, Egbert Eich wrote:
> Benjamin Herrenschmidt writes:
>  > I've though a lot about it, it's nasty...
>  > 
>  > The problem is that as soon as a card decodes legacy addresses, it must
>  > not be left enabled for IO/MEM accesses when another card needs to do
>  > VGA things.
>  > 
>  > I think what we need is an arbitrer with the following kind of APIs
>  > (which could be built on top of some sysfs manipulations & kenrel
>  > support or hidden in a library if kernel is out of the loop) :
>  > 
>  >  - vga_set_legacy_decoding(card, io_state, mem_state);
> 
> X uses an arbiter like this. We distinguish between different states:
> decoding_disabled, unused, used. Furthermore we distinguish between
> 'setup' and 'operating' state as some chips may need legacy resources
> for setup while during normal operation they may not be required.
> Also we treat 'memory' and 'io' resources separately.

Yes, I want to treat memory and io separately too. I think I don't need
to make a difference between setup and operating. As I wrote below, a
card that indicates that it doesn't do legacy decoding (safe) can still
use vga_get/put for a 'temporary' lock on the vga space, typically for
things like int10

> Based on this we decide which devices can stay enabled and which 
> driver instances need to call the arbiter before they are allowed to
> access resources.

Yup. One thing I want to avoid tho is callbacks to drivers, since it
becomes very difficult for a kernel based arbiter. So I think a driver
that calls vga_set_legacy_decoding() indicating it has set the card not
to decode legacy space can then safely not call the arbiter anymore
unless it needs temporary VGA lock for int10.

> Resources stay enabled as long as another device needs them. This
> reduces the number of switches.

Agreed, that's my intend too.
>  > 
>  >    This one would inform the arbitrer that your card is decoding VGA
>  > accesses or not. It should be called by the driver, by default, the
>  > arbitrer assumes all cards are decoding VGA addresses. I need to double
>  > check what is needed for radeon to stop decoding VGA accesses, I'm not
>  > 100% sure about that. CRTC_EXT_DISP_EN plus some other bit in DAC_CNTL
>  > may be enough, but we should make sure of that. I think nVidia's can
>  > remap the VGA IOs to some other place in PCI space with real decoding,
>  > so they become a non-issue.
> 
> I would like to come up with a solution that works with every card.
> The big breakthru of multihead setups came when we had this resource 
> manager (arbiter) in X which that was able to handle whatever cards
> were available on the market.

I would keep the arbiter API simple, and have the knowledge of the card
specific up to the card's driver tho. That is by default assume a card
does full legacy decoding unless the card's specific driver informs us
that this is not the case (typically because the driver disabled it on
the card).

>  > 
>  >    We should probably separate VGA IO and VGA Memory decoding, since
>  > some cards may be able to disable the later but not the former, but
>  > since a lot of cards can be used entirely without doing any IO, that can
>  > be dealt with (like radeons: you can entirely operate a radeon without
>  > using any IO access) in which case we can leave the IOs disabled in the
>  > PCI cmd register.
> 
> Right.
>  > 
>  >  - cookie = vga_get(card, io/mem);
>  > 
>  >    Request VGA IO and/or mem resources. Triggers disabling of other
>  > cards that have decoding of these enabled.
> 
> If we want to allow more than one instance (process) to access this we 
> may have to let this wait here until another instance has given up
> the resource. Currently we don't have to do this in X as everything 
> happens in a single thread.

Yes. I'm thinking in breaking this into a couple of versions in fact, a
blocking one and a non-blocking one (that just fails). Eventually a
blocking one with a timeout too.
>  > 
>  >  - vga_put(cookie);
>  > 
>  >    Release.
>  > 
>  > I don't think we need a callback to "clients" to inform them that they
>  > are losing VGA access, this would be to complicated to deal with as
>  > client can be separate processes and/or kernel land.
> 
> How about the interrupt case?

Explained later in the email. There is no way to deal cleanly with
interrupts other than only allowing them on a card that has permanent
access to resources, unless we want to introduce a callback mecanism for
"lost ownership". Possible, but complexifies the API. Do we really need
that ?

>  > For this to work, any VGA access must be bracketed by vga_get/vga_put.
>  > The implementation can be "lazy" (the actual cmd register switching is
>  > only done where an actual change of ownership is needed).
> 
> That's how we do it in the Xserver. We use a wrapper layer around any
> function that may access this.
> 
>  > 
>  > That also puts some restrictions on interrupts since I expect the above
>  > API to be only callable from normal process context (or it's impossible
>  > to deal with userland clients properly). It's not suitable to be called
>  > from an interrupt since vga_get may fail or block (to be decided, maybe
>  > we can have 2 versions...)
> 
> The interrupt version must do something to block the non-interrupt
> calls. 
> What if a second interrupt handler is called and serviced by another
> CPU while another handler is still active?

The API described above isn't for use by interrupts, but I explained it
later in the mail anyway.
>  > 
>  > Typically, a driver could only enable IRQ generation on the card in on
>  > those circumstances:
>  > 
>  >  - It does have VGA decoding enabled (it called vga_set_legacy_decoding
>  > with no IO and no MEM decoding set). It basically puts the card out of
>  > the arbitration domain.
> 
> Quite restrictive....

Yes, but is this a problem in practice ? We mostly need interrupts with
modern cards that do DRI. Can't they all disable legacy VGA decoding ?

If not, I'll have to come up with a callback based mecanism, so that the
current "owner" gets notified that it's about to lose it, so it can
disable interrupt generation on the card. I do _not_ intend to have an
API that allows switching the owner at interrupt time, that would be a
nightmare. If a driver really don't want to disable interrupts, it can
go the dirty way, and if it takes an interrupt while not the owner, just
call disable_irq() on it, trigger an asynchronous event (work queue)
and, from there, in process context, do the vga_get() to re-acquire
ownership and re-enable the IRQ. This will add latency to the interrupt
but I expect it to happen quite rarely.

I'm no fan of this solution tho, but if it's considered acceptable, then
the only API addition needed is a vga_tryget() that is callable at irq
time, that doesn't block and basically returns true if we still own the
VGA, or even eventually proceeds to the switch if nobody else has the
VGA semaphore held.

Such a tryget mecanism is needed anyway if we want to keep vgacon
ability to do printk at interrupt time.

>  >  - It does have VGA decoding enabled only for IO (decoding of the VGA
>  > memory aperture is disabled, that is it called vga_set_legacy_decoding
>  > with only IO set) and the interrupt handler doesn't need to do IO
>  > accesses. That is, it can afford to get interrupts while IO accesses are
>  > disabled in the config space (but not memory accesses).
> 
> OK.


>  >  - It hold the VGA semaphore (vga_get) but I don't recommend this
>  > scenario unless the driver knows the interrupt will come very soon
>  > (maybe suitable for vblank waiting).
> 
> Slighly better than polling...
>  > 
>  > Note that a driver that used vga_set_legacy_decoding(0,0) can still use
>  > vga_get()/vga_put() in case it needs to temporarily re-enable it, for
>  > things like POSTing. It must just make sure it re-disables it on the
>  > card before vga_put().
> 
> OK, at the moment it sounds reasonable.
>  >  
>  > Egbert, Jon, any comment ?
> 
> Our concept in X is more general however. We allow overlapping of any
> arbitrary resources. That's why we collect all resources from the devices.
> 
> Another set of leagacy resources are the 8514 registers which are 
> employed by the Mach64 driver (at least Mach64 decodes them).
> 8514 resources are a PITA as they are sparse and scattered
> accross PIO space.
> We've got all the code to handle that it would not be too difficult
> to extend it to general resource registration.

Hrm... I have to think about it. I could split, in my API, "IO" into
several classes of IOs, but i'd rather avoid adding too much complexity,
even if that ends up beeing at the expense of a little bit of perfs on
old card when multiple of them are used simultaneously.

>  > It could be implemented as a userland library for "default" OSes, that
>  > library using kernel facilities (to be written) on OSes that provide
>  > them (linux).
> 
> Right! It would help for older versions of Linux, too.
> 
>  > 
>  > As far as Linux is concerned, the toggling of VGA access with an
>  > in-kernel semaphore to be shared between userland toggle (via sysfs) and
>  > internal drivers (vgacon) has to be done, along with some indication (in
>  > sysfs too) of the decoding mode of the card as set by the driver (could
>  > be set by the kernel driver or by X, I suppose if they have different
>  > settings, they can switch it at VT switch time).
>  > 
> 
> OK.
> 
> Cheers,
> 	Egbert.
-- 
Benjamin Herrenschmidt <benh at kernel.crashing.org>