[systemd-devel] slowness reading persistent journal

[Vito Caputo]

On Wed, Apr 09, 2025 at 09:48:25AM +0000, Windl, Ulrich wrote:
> Hi!
> 
> In SLES15 SP6 (systemd-254.24-150600.4.28.1.x86_64) I noticed that collecting logs took quite long, so I checked.
> # time journalctl |wc -l
> 2705448
> 
> real    2m23.008s
> user    1m25.666s
> sys     0m57.389s
> 
> The journal is persistent and it's located on a BtrFS subvolume using a device that is provided by VMware from a high-performance SSD SAN storage:
> # lsblk
> NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
> sda      8:0    0   40G  0 disk
> ├─sda1   8:1    0  512M  0 part /boot/efi
> ├─sda2   8:2    0   21G  0 part /root
> │                               /srv
> │                               /usr/local
> │                               /var
> │                               /opt
> │                               /tmp
> │                               /boot/grub2/x86_64-efi
> │                               /boot/grub2/i386-pc
> │                               /.snapshots
> │                               /
> ├─sda3   8:3    0 10.7G  0 part /home
> └─sda4   8:4    0  7.8G  0 part [SWAP]
> # hdparm -t /dev/sda2
> 
> /dev/sda2:
> Timing buffered disk reads: 1680 MB in  3.00 seconds = 559.48 MB/sec
> # hdparm -T /dev/sda2
> 
> /dev/sda2:
> Timing cached reads:   18612 MB in  2.00 seconds = 9322.49 MB/sec
> 
> During boot this statistics had been logged:
> System Journal (/var/log/journal/41d6ce7a6cfc467297b777b8d87f071d) is 2.0G, max 2.1G, 103.1M free.
> 
> Obvious question: Why is it so slow?
> 

Not sure if it's impacting your scenario, but v257 got a fix for a
degenerate case in mmap-cache with commit 176f73.  I would try building
that or newer and seeing if its journalctl+dylibs does better on that
host.

- Vito