[Mesa-dev] Perfetto CPU/GPU tracing

Thu Feb 18 13:35:12 UTC 2021

Hi,

On Thu, 2021-02-18 at 12:17 +0100, Primiano Tucci wrote:
[discussion about Perfetto itself]
...
> eero.t.tamminen@
> from in common Linux distro repos.
> 
> That's right. I am aware of the problem. The plan is to address it
> with 
> bit.ly/perfetto-debian as a starter.

Glad to hear something is planned for making things easier for distros!

> > eero.t.tamminen@
> > Just set uprobe for suitable buffer swap function [1], and parse 
> kernel ftrace events. (paraphrasing for context: "why do we need 
> instrumentation points? we can't we just use uprobes instead?")
> 
> The problem with uprobes is:
> 1. It's linux specific. Perhaps not a big problem for Mesa here, but the
> reason why we didn't go there with Perfetto, at least until now, is that
> we need to support all major OSes (Linux, CrOS, Android, Windows,
> macOS).

The main point here is that uprobe works already without any
modifications to any libraries (I have script that has been used for FPS
tracking of daily 3D stack builds for many years).

And other OSes already offer similar functionality.  E.g. DTrace should
be available both for Mac & Windows.

Btw. If really needed, you could even implement similar functionality in
user space on other OSes.    We did something similar at Nokia using
ptrace before uprobes was a thing:
	https://github.com/maemo-tools-old/functracer

(While possible, because of the need for some tricky architecture
specific assembly, and certain instructions needing some extra assembly
fixups when replaced by a breakpoint, it's unlikely to be feasible for
general tracing though.)

> 2. Even on Linux-based systems, it's really hard to have uprobes enabled
> in production (I am not sure what is the situation for CrOS).
>
>  In Google, 
> we care a lot about being able to trace from production devices
> without 
> reflashing them with dev images, because then we can just tell people 
> that are experiencing problems "can you just open chrome://tracing, 
> orders of magnitude the actionable feedback we'd be able to get from 
> users / testers.

I would think having tracepoint code in the libraries themselves and
generally enabled for some specific tracing solution like Perfetto, is
*much* more unlikely.

At least both Ubuntu and Fedora default kernels have had uprobes built
in for *many* years.

Up to date Fedora 33 kernel:
$ grep UPROBE /boot/config-5.10.15-200.fc33.x86_64 
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y

Same on up to date Ubuntu 20.04:
$ grep UPROBE /boot/config-5.4.0-65-generic 
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y

> The challenge of ubprobes is that it relies on dynamic rewriting of 
> .text pages. Whenever I mention that, a platform security team reacts 
> like the Frau Blucher horses (https://youtu.be/bps5hJ5DQDw?t=10), with
> understandable reasons.

I'm not sure you've given them accurate picture of it.

It doesn't require executable code to be writable from user-space,
library code can remain read-only because kernel can toggle relevant
page writable for uprobe breakpoint setup and back.

# cat /sys/kernel/tracing/uprobe_events 
p:uprobes/glXSwapBuffers /opt/lib/libGL.so.1.2.0:0x000000000003bab0

# grep -h /opt/lib/libGL.so.1.2.0 /proc/*/maps | sort
7f486ab51000-7f486ab6a000 r--p 00000000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f486ab6a000-7f486abaf000 r-xp 00019000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f486abaf000-7f486abc6000 r--p 0005e000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f486abc6000-7f486abc9000 r--p 00074000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f486abc9000-7f486abca000 rw-p 00077000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f491438d000-7f49143a6000 r--p 00000000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f49143a6000-7f49143eb000 r-xp 00019000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f49143eb000-7f4914402000 r--p 0005e000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f4914402000-7f4914405000 r--p 00074000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f4914405000-7f4914406000 rw-p 00077000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
7f6296d62000-7f6296d7b000 r--p 00000000 08:03 7865435                  
/opt/lib/libGL.so.1.2.0
...

>  > mark.a.janes@
> events.  Why not follow the example of GPUVis, and write generic 
> trace_markers to ftrace?
> 
> In my experience ftrace's trace_marker:
> 1. Works for very simple types of events (e.g. 
> begin-slice/end-slice/counters) but don't work with richer / structured 
> event types like the ones linked above, as that gets into 
> stringification format, mashaling costs and interop.
> 2. Writing into the marker has some non-trivial cost (IIRC 1-10 us on 
> Android), it involves a kernel into and back from the kernel;
> 3. It leads to one global ring buffer, where fast events push out slower
> ones, which is particularly problematic if you ever enable sched/* 
> events. At the userspace level, instead, we can dynamically route 
> different events to different buffers to mitigate this problem (see 
> https://perfetto.dev/docs/concepts/config#dynamic-buffer-mapping)

Yes, if you need more context, or handle really frequent events, static
breakpoints are a better choice.

In case of more frequent events, on Linux one might consider using some
BPF program to process dynamic tracepoint data so that much smaller
amount needs to be transferred to user-space.  But I'm not sure whether
support for attaching BPF to tracepoints is in upstream Linux kernel
yet.

	- Eero