xrandr slow on elkhart lake debian (HDMI/DisplayPort)

[Dominik Wrona]

I have a bash script which iterates over xinput entries and assigns 3 
USB touchscreen inputs to 3 respective displays using xinput 
map-to-output. This command is super slow (we're talking about 30+ 
seconds to complete), whereas on other processors and setups this can be 
done in a second or less.
I used to think the issue was with xinput, however, other xinput 
commands work decently quick, whereas it is typing xrandr where I 
noticed a similar, though shorter, lag. So the issue is with the 
connection to the monitors.
I am using DisplayPort ports, with Benfei DisplayPort-to-HDMI adapters 
to connect to 3 HDMI displays,  on a debian elkhart lake intel processor.
Is there anything I can do to make commands affecting the displays run 
normally, without this abnormal lag? There is zero lag on the displays 
themselves, so the issue seems more with processing the display input 
than the data being sent on the cable itself.

Regards,
Dominik