Some HDMI deep color output fixes for DC on DCE 6-11

[Mario Kleiner]

Hi,

these two patches fix non-working HDMI deep color output on DCE-8.3,
AMD Mullins when amdgpu-kms is used with Displaycore force-enabled,
ie. for radeon.cik_support=0 amdgpu.cik_support=1 amdgpu.dc=1:
I suspect they might fix similar problems on other older asics of
DCE-11.0 and earlier.

Patch 1/2 is a fix for some oddity i found while hunting for the
HDMI deep color bug. It fixes what looks like an obvious mistake,
but the fix did not improve or degrade anything, so maybe the hw
doesn't care all too much about the wrong clamping/truncation mask?
Anyway, it makes the code less confusing.

Patch 2/2 fixes HDMI deep color output at 10 bpc or 12 bpc output
on AMD Mullins, DCE-8.3, where at output_bpc 10 or 12 the display
would be scrambled. With the patch, the display looks correct, and
photometer measurements on a HDR-10 monitor suggest we probably get
the correct output signal. I found the fix by comparing DC against
the classic amdgpu-kms and radeon-kms modesetting path, readding
missing stuff.

Given other encoder/pixelclock setup functions than the ones used
on DCE-8.3 showed the same omissions, i added missing code there as
well, but i couldn't test it due to lack of hw, so i hope that fixes
instead of breaks things on asic's other than DCE-8.3.

I also created a similar patch for DCE-11.2 and later, not included
here, but during testing on a Raven Ridge DCN-1, the patch neither
helped nor hurt. Output was correct without the patch, and adding the
patch didn't change or break anything on DCN-1. Looking at disassembled
AtomBios tables for DCN-1 and a DCE-11.2, i think AtomBios may not do
much with the info that was missing, which would explain why the
current upstream code seems to work fine without it? At least as
verified on DCN-1. I can't test on DCE-11.2 or DCE-12 due to lack
of hw with actual HDMI output. But it would be interesting for me to
know what changed wrt. Atombios in later asic versions to make some
of this setup apparently redundant in DC?

Do you test DC wrt. HDMI deep color starting at a specific DCE
revision, given that the bug went unnoticed in DCE-8.3, but things
seem to be fine on at least DCN-1?

Thanks,
-mario