[systemd-devel] Storing package metadata in ELF objects

[Luca Boccassi]

On Wed, 2021-05-19 at 16:58 +0200, Mark Wielaard wrote:
> Hi Guillem,
> 
> On Wed, 2021-05-19 at 02:19 +0200, Guillem Jover wrote:
> > So this is where I guess I'm missing something. To be able to make
> > sense of the coredumps there are two things that might end up being
> > relevant, backtraces and source code. systemd-coredump might already
> > emit a backtrace, and depending on the information provided it might
> > be more or less useful. If one needs the actual debug symbols there's
> > already some external querying/fetching required, and if distribution
> > specific source code is required because many distributions patch
> > upstream source, then even more querying/fetching will be required.
> > 
> > Which is why I'm not seeing why this standalone and isolated metadata
> > would be of much help by itself. As in, the way I see it, either the
> > information from systemd (w/o the extra metadata) is sufficient to
> > track down bugs, or that querying/fetching would be needed anyway, at
> > which point the metadata can be inferred too then?
> 
> Because without that metadata you cannot easily figure out where/how to
> get the files needed to properly track down the bugs. But using that
> metadata you can figure out where the debuginfod server is that can
> provide all that information for the binaries/core files (or a distro
> specific method if the distributor doesn't have a debuginfod server
> yet).

Yes, that is a good use case. Another use case is that the person (or
bot) who first looks at the journal with the crash notification and the
metadata might not necessarily be the one that does the full in-depth
debugging. First level support will look at things and go "oh this is
from package X at version Y, therefore team Z needs to chime in".
Or it can see that it's version A.B.C, which is listed as known buggy,
and ignore the issue. Or a myriad of other combinations.

Then there are automated systems, parsing logs and creating tickets.
Having the structured metadata available in the journal in the crash
message means much more complex and useful querying and ticket creation
systems can be set up. You _cannot_ just go and query random remote
internet servers from these systems, they need to be network-isolated,
as logs can have confidential data from customers.

The main point is, there are many many things that happen between a
crash and the team owning the component looking up debugging symbols
and loading core files. This feature is very very useful for a whole
lot of those things.

> > Oh, I was thinking about those mixed environments, full chroots or
> > stripped down containers, from different vendors, but affecting Debian
> > installations. What I'm also probably missing here is how does the
> > metadata help for a third-party that is not expected to track/keep/upload
> > debug symbols nor source packages into some repository, because otherwise
> > I'm not seeing how they'd make use of the cores (if they are insufficient
> > by themselves) to debug issues? I guess my question back would be what
> > would they do with the metadata if they do not have the debug symbols
> > nor the sources readily available? Also assuming of course they exercise
> > good practices such as never reusing the same package-version-arch tuple
> > for different builds, and similar. :)
> 
> Different builds will have different build-ids, so they can be kept
> apart. But even if without the distributor having added debuginfod
> meta-data and providing a server to fetch the extra symbols, debuginfo,
> sources, the extra meta-data is useful to administrators. Just seeing
> that crashes are associated with specific distro/version/packages helps
> narrow down instabilities.

Precisely - again this is a very much real use case. In my case, the
engineers doing support and looking at the logs (and only at the logs -
no access to the running systems is permitted) _want_ this, because it
helps them. Being able to know at a glance to the journal exactly what
is borken, with version info, is extremely valuable to them.

Yes the version info might not be precise for a minority of use cases
that override the binary version with something different than the
source version, but that's fine as it's far and few, mostly affects
metapackages, and even then it can be documented clearly that the
reference is to the source version, not the binary one.

-- 
Kind regards,
Luca Boccassi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20210524/2617c81f/attachment.sig>