Must-Pass Test Suite for KMS drivers

[Rob Clark]

On Mon, Oct 24, 2022 at 5:43 AM <maxime at cerno.tech> wrote:
>
> Hi,
>
> I've discussing the idea for the past year to add an IGT test suite that
> all well-behaved KMS drivers must pass.
>
> The main idea behind it comes from v4l2-compliance and cec-compliance,
> that are being used to validate that the drivers are sane.
>
> We should probably start building up the test list, and eventually
> mandate that all tests pass for all the new KMS drivers we would merge
> in the kernel, and be run by KCi or similar.

Let's get https://patchwork.freedesktop.org/patch/502641/ merged
first, that already gives us a mechanism similar to what we use in
mesa to track pass/fail/flake

Beyond that, I think some of the igt tests need to get more stable
before we could consider a "mustpass" list.  The kms_lease tests seem
to fail on msm due to bad assumptions in the test about which CRTCs
primary planes can attach to.  The legacy-cursor crc tests seem a bit
racy (there was a patch posted for that, not sure if it landed yet),
etc.

The best thing to do is actually start running CI and tracking xfails
and flakes ;-)

BR,
-R

> I did a first pass to create a draft of such a test-suite, which would
> contain:
>
> igt at core_auth@basic-auth
> igt at core_auth@getclient-master-drop
> igt at core_auth@getclient-simple
> igt at core_auth@many-magics
> igt at core_getclient
> igt at core_getstats
> igt at core_getversion
> igt at core_hotunplug@hotrebind-lateclose
> igt at core_hotunplug@hotunbind-rebind
> igt at core_hotunplug@unbind-rebind
> igt at core_setmaster
> igt at core_setmaster_vs_auth
> igt at device_reset@unbind-reset-rebind
> igt at drm_read
> igt at dumb_buffer
> igt at fbdev
> igt at feature_discovery@display
> igt at kms_3d
> igt at kms_addfb_basic
> igt at kms_async_flips
> igt at kms_color
> igt at kms_concurrent
> igt at kms_cursor_crc
> igt at kms_cursor_edge_walk
> igt at kms_cursor_legacy@basic-busy-flip-before-cursor
> igt at kms_cursor_legacy@basic-flip-after-cursor
> igt at kms_cursor_legacy@basic-flip-after-cursor
> igt at kms_display_modes
> igt at kms_dither
> igt at kms_dp_aux_dev
> igt at kms_flip@basic-flip-vs-dpms
> igt at kms_flip@basic-flip-vs-modeset
> igt at kms_flip@basic-flip-vs-wf_vblank
> igt at kms_flip@basic-plain-flip
> igt at kms_flip_event_leak@basic
> igt at kms_force_connector_basic@force-connector-state
> igt at kms_force_connector_basic@force-edid
> igt at kms_force_connector_basic@force-load-detect
> igt at kms_force_connector_basic@prune-stale-modes
> igt at kms_getfb
> igt at kms_hdmi_inject
> igt at kms_hdr
> igt at kms_invalid_mode
> igt at kms_lease
> igt at kms_panel_fitting
> igt at kms_pipe_crc_basic
> igt at kms_plane_alpha_blend
> igt at kms_plane
> igt at kms_plane_cursor
> igt at kms_plane_lowres
> igt at kms_plane_multiple
> igt at kms_plane_scaling
> igt at kms_prop_blob
> igt at kms_properties
> igt at kms_rmfb
> igt at kms_scaling_modes
> igt at kms_sequence
> igt at kms_setmode
> igt at kms_sysfs_edid_timing
> igt at kms_tv_load_detect
> igt at kms_universal_plane
> igt at kms_vblank
> igt at kms_vrr
> igt at kms_writeback
>
> Most of them are skipped on vc4 right now, but I could see that some of
> them fail already (kms_rmfb, core_hotunplug), so it proves to be useful
> already.
>
> What do you think? Is there some more tests needed, or did I include
> some tests that shouldn't have been there?
>
> Thanks!
> Maxime