[PATCH 0/2] drm: revert some framebuffer API tests

[Guenter Roeck]

On 9/24/24 08:56, Jani Nikula wrote:
> On Tue, 24 Sep 2024, Guenter Roeck <linux at roeck-us.net> wrote:
>>>>> On Tue, Sep 24, 2024 at 12:06:28PM GMT, Simona Vetter wrote:
>>>>>> Yeah I think long-term we might want a kunit framework so that we can
>>>>>> catch dmesg warnings we expect and test for those, without those warnings
>>>>>> actually going to dmesg. Similar to how the lockdep tests also reroute
>>>>>> locking validation, so that the expected positive tests don't wreak
>>>>>> lockdep for real.
>>>>>>
>>>>>> But until that exists, we can't have tests that splat in dmesg when they
>>>>>> work as intended.
>>
>> FWIW, that is arguable. More and more tests are added which do add such splats,
>> and I don't see any hesitance by developers to adding more. So far I counted
>> two alone in this commit window, and that does not include new splats from
>> tests which I had already disabled. I simply disable those tests or don't
>> enable them in the first place if they are new. I did the same with the drm
>> unit tests due to the splats generated by the scaling unit tests, so any
>> additional drm unit test splats don't make a difference for me since the
>> tests are already disabled.
> 
> What's the point of having unit tests that CI systems routinely have to
> filter out of test runs? Or filter warnings generated by the tests,
> potentially missing new warnings. Who is going to run the tests if the
> existing CI systems choose to ignore them?
> 
> Automation on a massive scale is key here, and making that harder is
> counter-productive.
> 

As I have said elsewhere, not being able to handle backtraces generated
on purpose is a limitation of my testbed. Other testbeds with active (and
paid) maintainers may not have that limitation. I deal with it by disabling
affected tests. Others may deal with it by maintaining exception lists.

Ultimately it is up to developers and testbed maintainers to decide the level
of test coverage they are looking for or able to support.

Thanks,
Guenter