Why is Thunderbolt 3 limited to 2.5 GT/s on Linux?

[Alex Deucher]

On Mon, Jul 1, 2019 at 10:38 AM Timur Kristóf <timur.kristof at gmail.com> wrote:
>
> > > > > Like I said the device really is limited to 2.5 GT/s even
> > > > > though it
> > > > > should be able to do 8 GT/s.
> > > >
> > > > There is Thunderbolt link between the host router (your host
> > > > system)
> > > > and
> > > > the eGPU box. That link is not limited to 2.5 GT/s so even if the
> > > > slot
> > > > claims it is PCI gen1 the actual bandwidth can be much higher
> > > > because
> > > > of
> > > > the virtual link.
> > >
> > > Not sure I understand correctly, are you saying that TB3 can do 40
> > > Gbit/sec even though the kernel thinks it can only do 8 Gbit / sec?
> > >
> > > I haven't found a good way to measure the maximum PCIe throughput
> > > between the CPU and GPU, but I did take a look at AMD's sysfs
> > > interface
> > > at /sys/class/drm/card1/device/pcie_bw which while running the
> > > bottlenecked game. The highest throughput I saw there was only 2.43
> > > Gbit /sec.
> > >
> > > One more thought. I've also looked at
> > > /sys/class/drm/card1/device/pp_dpm_pcie - which tells me that
> > > amdgpu
> > > thinks it is running on a 2.5GT/s x8 link (as opposed to the
> > > expected 8
> > > GT/s x4). Can this be a problem?
> >
> > We limit the speed of the link the the driver to the max speed of any
> > upstream links.  So if there are any links upstream limited to 2.5
> > GT/s, it doesn't make sense to clock the local link up to faster
> > speeds.
> >
> > Alex
>
> Hi Alex,
>
> I have two concerns about it:
>
> 1. Why does amdgpu think that the link has 8 lanes, when it only has 4?

Not sure.  We use pcie_bandwidth_available() on kernel 5.3 and newer
to determine the max speed and links when we set up the GPU.  For
older kernels use use an open coded version of it in the driver.

>
> 2. As far as I understood what Mika said, there isn't really a 2.5 GT/s
> limitation there, since the virtual link should be running at 40 Gb/s
> regardless of the reported speed of that device. Would it be possible
> to run the AMD GPU at 8 GT/s in this case?

If there is really a faster link here then we need some way to pass
that information to the drivers.  We rely on the information from the
upstream bridges and the pcie core helper functions.

Alex