Why is Thunderbolt 3 limited to 2.5 GT/s on Linux?

[Marek Olšák]

You can run:
AMD_DEBUG=testdmaperf glxgears

It tests transfer sizes of up to 128 MB, and it tests ~60 slightly
different methods of transfering data.

Marek

On Wed, Jul 3, 2019 at 4:07 AM Michel Dänzer <michel at daenzer.net> wrote:

> On 2019-07-02 11:49 a.m., Timur Kristóf wrote:
> > On Tue, 2019-07-02 at 10:09 +0200, Michel Dänzer wrote:
> >> On 2019-07-01 6:01 p.m., Timur Kristóf wrote:
> >>> On Mon, 2019-07-01 at 16:54 +0200, Michel Dänzer wrote:
> >>>> On 2019-06-28 2:21 p.m., Timur Kristóf wrote:
> >>>>> I haven't found a good way to measure the maximum PCIe
> >>>>> throughput
> >>>>> between the CPU and GPU,
> >>>>
> >>>> amdgpu.benchmark=3
> >>>>
> >>>> on the kernel command line will measure throughput for various
> >>>> transfer
> >>>> sizes during driver initialization.
> >>>
> >>> Thanks, I will definitely try that.
> >>> Is this the only way to do this, or is there a way to benchmark it
> >>> after it already booted?
> >>
> >> The former. At least in theory, it's possible to unload the amdgpu
> >> module while nothing is using it, then load it again.
> >
> > Okay, so I booted my system with amdgpu.benchmark=3
> > You can find the full dmesg log here: https://pastebin.com/zN9FYGw4
> >
> > The result is between 1-5 Gbit / sec depending on the transfer size
> > (the higher the better), which corresponds to neither the 8 Gbit / sec
> > that the kernel thinks it is limited to, nor the 20 Gbit / sec which I
> > measured earlier with pcie_bw.
>
> 5 Gbit/s throughput could be consistent with 8 Gbit/s theoretical
> bandwidth, due to various overhead.
>
>
> > Since pcie_bw only shows the maximum PCIe packet size (and not the
> > actual size), could it be that it's so inaccurate that the 20 Gbit /
> > sec is a fluke?
>
> Seems likely or at least plausible.
>
>
> >>>>> but I did take a look at AMD's sysfs interface at
> >>>>> /sys/class/drm/card1/device/pcie_bw which while running the
> >>>>> bottlenecked
> >>>>> game. The highest throughput I saw there was only 2.43 Gbit
> >>>>> /sec.
> >>>>
> >>>> PCIe bandwidth generally isn't a bottleneck for games, since they
> >>>> don't
> >>>> constantly transfer large data volumes across PCIe, but store
> >>>> them in
> >>>> the GPU's local VRAM, which is connected at much higher
> >>>> bandwidth.
> >>>
> >>> There are reasons why I think the problem is the bandwidth:
> >>> 1. The same issues don't happen when the GPU is not used with a TB3
> >>> enclosure.
> >>> 2. In case of radeonsi, the problem was mitigated once Marek's SDMA
> >>> patch was merged, which hugely reduces the PCIe bandwidth use.
> >>> 3. In less optimized cases (for example D9VK), the problem is still
> >>> very noticable.
> >>
> >> However, since you saw as much as ~20 Gbit/s under different
> >> circumstances, the 2.43 Gbit/s used by this game clearly isn't a hard
> >> limit; there must be other limiting factors.
> >
> > There may be other factors, yes. I can't offer a good explanation on
> > what exactly is happening, but it's pretty clear that amdgpu can't take
> > full advantage of the TB3 link, so it seemed like a good idea to start
> > investigating this first.
>
> Yeah, actually it would be consistent with ~16-32 KB granularity
> transfers based on your measurements above, which is plausible. So
> making sure that the driver doesn't artificially limit the PCIe
> bandwidth might indeed help.
>
> OTOH this also indicates a similar potential for improvement by using
> larger transfers in Mesa and/or the kernel.
>
>
> --
> Earthling Michel Dänzer               |              https://www.amd.com
> Libre software enthusiast             |             Mesa and X developer
> _______________________________________________
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20190703/9c4ad999/attachment.html>