[systemd-devel] How can we debug systemd-gpt-auto-generator failures?

[Kevin P. Fleming]

Thanks for that, it did indeed produce some output, but unfortunately
it doesn't seem to lead anywhere specific :-)

root at edge21-a:~# SYSTEMD_LOG_LEVEL=debug SYSTEMD_LOG_TARGET=console
LIBBLKID_DEBUG=all
/usr/lib/systemd/system-generators/systemd-gpt-auto-generator
Found container virtualization none.
Disabling root partition auto-detection, root= is defined.
Disabling root partition auto-detection, root= is defined.
Failed to open device: No such device

Adding strace to the command provides something more useful:

openat(AT_FDCWD, "/", O_RDONLY|O_CLOEXEC|O_PATH|O_DIRECTORY) = 3
openat(3, "sys", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
fstat(4, {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
close(3)                                = 0
openat(4, "dev", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
close(4)                                = 0
openat(3, "block", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
fstat(4, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
close(3)                                = 0
openat(4, "0:0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = -1 ENOENT (No
such file or directory)
close(4)

So it's trying to open() /sys/dev/block/0:0, but my system does not
have that device file. The only files in /sys/dev/block are 8:0
through 8:3.

On Thu, Jul 28, 2022 at 7:17 AM Mantas Mikulėnas <grawity at gmail.com> wrote:
>
> On Thu, Jul 28, 2022 at 1:51 PM Kevin P. Fleming <kevin at km6g.us> wrote:
>>
>> I've got two systems that report a failure (exit code 1) every time
>> systemd-gpt-auto-generator is run. There are a small number of reports
>> of this affecting other users too:
>>
>> https://bugs.archlinux.org/task/73168
>>
>> This *may* be related to the use of ZFS, although I've got a
>> half-dozen systems using ZFS and only two of them have this issue.
>>
>> Running the generator from the command line also produces exit code 1,
>> but doesn't produce any output. is there any practical way to debug
>> this failure?
>
>
> Start with SYSTEMD_LOG_LEVEL=debug (and SYSTEMD_LOG_TARGET=console if you want to run the tool from CLI, otherwise it logs to journal). If it only fails as part of a daemon-reexec, `systemctl set-environment` (not sure if `systemctl log-level` has any effect on non-pid1 processes). Since it uses libblkid, there's also LIBBLKID_DEBUG=all.
>
> --
> Mantas Mikulėnas