[igt-dev] Must-Pass Test Suite for KMS drivers

[Juha-Pekka Heikkila]

On 24.10.2022 18.58, Ville Syrjälä wrote:
> On Mon, Oct 24, 2022 at 08:48:15AM -0700, Rob Clark wrote:
>> On Mon, Oct 24, 2022 at 5:43 AM <maxime at cerno.tech> wrote:
>>>
>>> Hi,
>>>
>>> I've discussing the idea for the past year to add an IGT test suite that
>>> all well-behaved KMS drivers must pass.
>>>
>>> The main idea behind it comes from v4l2-compliance and cec-compliance,
>>> that are being used to validate that the drivers are sane.
>>>
>>> We should probably start building up the test list, and eventually
>>> mandate that all tests pass for all the new KMS drivers we would merge
>>> in the kernel, and be run by KCi or similar.
>>
>> Let's get https://patchwork.freedesktop.org/patch/502641/ merged
>> first, that already gives us a mechanism similar to what we use in
>> mesa to track pass/fail/flake
>>
>> Beyond that, I think some of the igt tests need to get more stable
>> before we could consider a "mustpass" list.  The kms_lease tests seem
>> to fail on msm due to bad assumptions in the test about which CRTCs
>> primary planes can attach to.  The legacy-cursor crc tests seem a bit
>> racy (there was a patch posted for that, not sure if it landed yet),
>> etc.
> 
> I think the safest set to start with would be pure uapi validation
> stuff. Anything that interactics with real world hardware is a much
> tougher cookie.
> 

I agree with Ville

As is with different pixel formats on different kms crc tests there are 
specialities just to make crcs happy hence I think crc tests will need 
to be carefully chosen for this type test set.

And as for those legacy cursor tests to be included people first need 
consensus across drivers how those tests are supposed to work and then 
strip out platform specific quirks from those tests.

/Juha-Pekka

>>
>> The best thing to do is actually start running CI and tracking xfails
>> and flakes ;-)
>>
>> BR,
>> -R
>>
>>> I did a first pass to create a draft of such a test-suite, which would
>>> contain:
>>>
>>> igt at core_auth@basic-auth
>>> igt at core_auth@getclient-master-drop
>>> igt at core_auth@getclient-simple
>>> igt at core_auth@many-magics
>>> igt at core_getclient
>>> igt at core_getstats
>>> igt at core_getversion
>>> igt at core_hotunplug@hotrebind-lateclose
>>> igt at core_hotunplug@hotunbind-rebind
>>> igt at core_hotunplug@unbind-rebind
>>> igt at core_setmaster
>>> igt at core_setmaster_vs_auth
>>> igt at device_reset@unbind-reset-rebind
>>> igt at drm_read
>>> igt at dumb_buffer
>>> igt at fbdev
>>> igt at feature_discovery@display
>>> igt at kms_3d
>>> igt at kms_addfb_basic
>>> igt at kms_async_flips
>>> igt at kms_color
>>> igt at kms_concurrent
>>> igt at kms_cursor_crc
>>> igt at kms_cursor_edge_walk
>>> igt at kms_cursor_legacy@basic-busy-flip-before-cursor
>>> igt at kms_cursor_legacy@basic-flip-after-cursor
>>> igt at kms_cursor_legacy@basic-flip-after-cursor
>>> igt at kms_display_modes
>>> igt at kms_dither
>>> igt at kms_dp_aux_dev
>>> igt at kms_flip@basic-flip-vs-dpms
>>> igt at kms_flip@basic-flip-vs-modeset
>>> igt at kms_flip@basic-flip-vs-wf_vblank
>>> igt at kms_flip@basic-plain-flip
>>> igt at kms_flip_event_leak@basic
>>> igt at kms_force_connector_basic@force-connector-state
>>> igt at kms_force_connector_basic@force-edid
>>> igt at kms_force_connector_basic@force-load-detect
>>> igt at kms_force_connector_basic@prune-stale-modes
>>> igt at kms_getfb
>>> igt at kms_hdmi_inject
>>> igt at kms_hdr
>>> igt at kms_invalid_mode
>>> igt at kms_lease
>>> igt at kms_panel_fitting
>>> igt at kms_pipe_crc_basic
>>> igt at kms_plane_alpha_blend
>>> igt at kms_plane
>>> igt at kms_plane_cursor
>>> igt at kms_plane_lowres
>>> igt at kms_plane_multiple
>>> igt at kms_plane_scaling
>>> igt at kms_prop_blob
>>> igt at kms_properties
>>> igt at kms_rmfb
>>> igt at kms_scaling_modes
>>> igt at kms_sequence
>>> igt at kms_setmode
>>> igt at kms_sysfs_edid_timing
>>> igt at kms_tv_load_detect
>>> igt at kms_universal_plane
>>> igt at kms_vblank
>>> igt at kms_vrr
>>> igt at kms_writeback
>>>
>>> Most of them are skipped on vc4 right now, but I could see that some of
>>> them fail already (kms_rmfb, core_hotunplug), so it proves to be useful
>>> already.
>>>
>>> What do you think? Is there some more tests needed, or did I include
>>> some tests that shouldn't have been there?
>>>
>>> Thanks!
>>> Maxime
>