[REGRESSION] amdgpu enables DSC unnecessarily over MST hubs on kernel 6.11+

[Peter Cai]

Hi,

I have been using a DisplayPort MST hub with a AMD Ryzen 6800U laptop 
(GPD Win Max 2) for a while. The hub exposes 2 downstream ports, both of 
which are connected to 4K (3840x2160) displays. Recently I was hit with 
another regression in Linux 6.10 (link: 
https://lore.kernel.org/stable/d74a7768e957e6ce88c27a5bece0c64dff132e24@holm.dev/T/#u) 
that caused MST displays to stop lighting up. This was caused by 
4df96ba66760345471a85ef7bb29e1cd4e956057 and later fixed in mainline by 
fa57924c76d995e87ca3533ec60d1d5e55769a27.

However, even with that fix, one of my displays is still misbehaving. 
The display seems to be running at no more than 20Hz even though the 
output mode is clearly set to 3840x2160 at 60 from the point of view of 
both the Wayland compositor and the kernel. Some debugging again pointed 
to the refactored `dm_dp_mst_is_port_support_mode()` function involved 
in the aforementioned commits as the problem. Specifically, it seems 
that the function before commit 4df96ba66760345471a85ef7bb29e1cd4e956057 
returns `DC_OK` for my setup, while the version after 
`fa57924c76d995e87ca3533ec60d1d5e55769a27` insists that it must set the 
DSC bit, shown by the debug kernel message "Require dsc and dsc config 
found". When the DSC flag is set, this display of mine seems to just 
misbehave like what is mentioned.

I've done some further print-based debugging to identify why the DSC 
branch is now taken. Adding a print right before

 > if (stream_kbps <= end_to_end_bw_in_kbps) {

gives me

 > [   38.079643] [drm:dm_dp_mst_is_port_support_mode [amdgpu]] 
stream_kbps = 12798000, root_link_bw_in_kbps = 12571200, 
virtual_channel_bw_in_kbps = 17176320

Since `end_to_end_bw_in_kbps` is `min(root_link_bw_in_kbps, 
virtual_channel_bw_in_kbps)`, this causes the DSC branch to be taken.

Before the changes that took `root_link_bw_in_kbps` into consideration, 
my setup was clearly working and I haven't encountered any problem 
whatsoever. The value `12571200` also feels a bit off to me since if 
this is fetched from `aconnector->dc_link`, it should represent the 
bandwidth available on the entire DP link between PC <-> MST Hub? In 
this case, it seems to be way too low to be correct; and, again, the 
dual 4K display setup has been working without checking this value. 
Modifying the latest mainline to remove `root_link_bw_in_kbps` from the 
equation also seems to fix the problem for me.

It feels like my display shouldn't misbehave like that with the DSC bit 
set, but again it is the kernel that didn't set the DSC bit before but 
now does. I think this should count as a regression, but I'm not sure if 
it is just me who is seeing this problem.

For reference, the problematic display in question here is a Samsung 
LS27A80. The MST hub used is an Anker 554 USB-C Docking Station / KVM 
Switch.

Regards,
Peter.