[PATCH] drm: Shift wrap bug in create_in_format_blob()

[Dan Carpenter]

On Wed, Aug 09, 2017 at 03:38:33PM +0100, Daniel Stone wrote:
> On 9 August 2017 at 15:36, Sean Paul <seanpaul at chromium.org> wrote:
> > On Wed, Aug 09, 2017 at 02:19:06PM +0300, Dan Carpenter wrote:
> >> "plane->format_count" can go up to 64.  (It's capped in
> >> drm_universal_plane_init().)  So we should be using ULL type instead of
> >> int here to prevent shift wrapping.
> >>
> >> Fixes: db1689aa61bd ("drm: Create a format/modifier blob")
> >> Signed-off-by: Dan Carpenter <dan.carpenter at oracle.com>
> >
> > Thank you for the fix, Dan.
> >
> > I've applied it to drm-misc-next.
> 
> Yes, thanks Dan!
> 
> Out of interest, how was this found? With sparse?
> 

These are Smatch checks that I haven't totally cleaned up enough to
publish yet.  I have a couple versions of this check.  This one is doing
cross function analysis so it knows that ->format_count can go up to 64
bits.

regards,
dan carpenter