[PATCH RFC] dt-bindings: display: document display panel occlusions

[Caleb Connolly]

On 10/10/2023 21:01, Piotr Masłowski wrote:
> Hi Caleb,
> 
> Thanks for posting this. I've been meaning to chime in on the discussion
> about notches and co. for months now, so this makes a perfect opportunity
> to finally do so.
> 
> On Mon Oct 9, 2023 at 7:32 PM CEST, Caleb Connolly wrote:
> 
>> Some folks have previously suggested that this information belongs in
>> userspace and not in devicetree. I would like to be clear that
>> devicetree is for describing hardware, and parts of a display which can
>> never actually be seen are very much properties of the underlying
>> hardware.
> 
> Yes, thank you! Seeing the kernel log (or Tuxes) partially hidden behind
> the notch (or the top-left rounded corner) is just so annoying, so DT is
> definitely the way to go here.


Exactly! Glad that others care about the _real_ issue here ;P

[...]
> 
> 
> So why am I writing all of this? Well, the problem I see is that any
> shape-based approach will likely suffer from both accuracy and
> complexity issues. Describing curves is hard and processing them is
> not something that should be included in e.g. whatever handles VTs.

My proposal here doesn't require processing curves or doing any
complicated calculations. If you know that the display has a 30px radius
on all corners, you can adjust the VT to not use the top 30px of the
screen and it will start exactly where the radius stops.

If you know that there is a (potentially very curvy) notch at the top of
the screen, you can just iterate over the arcs, add their radius to
their Y coordinate and take the maximum. This will always give you at
least the height of the notch (you'd probably want to check that their
angle is vaguely downward, but again this is trivial fast integer math).

[...]
> 
> However, there's a different approach that is both extremely simple and
> yet trivially correct – pixel masks!
> 
> Basically, instead of trying to describe what shape a cutout, notch or
> other feature has, we just say which pixels belong to it. In short, this:
> 
> - can be easily processed even by 'dumb' programs – for example, you can
>   derive minimal margins to 'letterbox' the screen

As above (and as mentioned in the patch) this is also trivial to derive
from the curves.
> - gathering the data is very straightforward – just light the relevant
>   pixels one-by-one and check if you see them
> - pixel-perfect accuracy is the default

I think it would be fairly straightforward to do this for curve data
too. You just bump the radius up/down until it looks right, or "good enough"
> 
> Of course this is not a perfect solution. Here are the drawback that I
> can see:
> 
> - low resolution screens mean low resolution data
> - 'smart' programs may or may not need a bit more logic than otherwise
> - sub-pixel accuracy is impossible or requires extending this scheme
>   - non-binary (fractional) weights
>   - partially-occluded pixels on a separate mask (so each feature would
>     have two adjacent masks – for its interior and for the border)
> 
> 
> As a futher improvement, besides looking at pixels we could think about
> subpixels instead. Still, this can easily be added later and likely even
> in a backwards-compatible manner.
> 
> An orthogonal issue is labeling all of those regions. I think we should
> start with fully obscured areas and maybe less readable ones like the
> waterfall edges. Still, different features should live on different
> masks – even if we don't attach meaningfull labels (like 'notch' or
> 'camera cutout') to them right away.
> 
> 
> What do you all think of that? I didn't see this approach considered in
> any of the earlier discussions, yet it seems so elegant to me. Am I
> missing something?

I think the unfortunate truth is that approximating notches and rounded
corners exclusively with regular arcs at the cost of pixel accuracy is
just such a no-brainer. Pixel masks would be pixel accurate, but there
is no benefit compared to a slightly underfit curve.

Any program which wanted to make use of the curve information would have
to run a whole curve fitting algorithm, whereas there simply aren't any
programs which need to know about display occlusions to a pixel-accurate
level. For padding a status bar you do a single trig equation, for
avoiding the notch in fullscreen you would query the DRM subsystem which
presumably would export the dimensions of a letterboxed "usable size".

To be clear though, I wouldn't mind being proven wrong on this, but I'm
yet to find some information which doesn't line up with this conclusion.
> 
> --
> Cheers,
> Piotr Masłowski

-- 
// Caleb (they/them)