[systemd-devel] odd behaviour on corrupted journal/further analysis

Mon Feb 19 03:19:11 UTC 2024

Dominique Martinet wrote on Sun, Feb 18, 2024 at 06:43:25PM +0900:
> Anyway, my main question stays there same: I couldn't find any
> diagnostic utility; I don't care if I just get a garbled mess but is
> there a way to try to decompress what we can or whatever is needed to
> get infos out of the corrupted files? Or is journalctl already doing as
> much as is possible?
> I've checked the files with xxd and they're not empty, I can't tell if
> it's random garbage or if there's anything useful in here but would be
> happy to send a couple of files to a few individuals (probably nothing
> too sensible in there, but would rather not make it public as it
> might contain some mail addresses or whatsnot)

Hmpf. So I had copied /var/log/journal to preserve it just in case, but
didn't try reading from it yesterday...

Now I've tried `journalctl -D copied_dir` it's finding most of the old
logs just fine up till the second the system became unresponsive, so I
was able to get as much logs as I could hope for here.

That just leaves a real bug of why it doesn't work directly in
/var/log/journal -- I expect it'll take a few months for the bad file to
rotate out if it ever does, so I guess I'll open a github issue with the
same details I sent yesterday here when I have a moment... Might take a
week or so, so feel free to request extra details or anything here
directly if someone has an idea.

For anyone who cares: running `journalctl -D . --verify` in that
directory properly checks all the old files; there is also one file that
is ignored from the start and one file that is corrupted upon
verification but it's fairly different from the output I sent yesterday.
I've attached debug output for completeness.

Cheers,
-- 
Dominique Martinet | Asmadeus
-------------- next part --------------
Considering root directory '.'.
Root directory . added.
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b531a7b-00060f1593b6379e.journal: Bad message
Failed to open journal file ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b53b7f5-00060f1fcebd0c6a.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b53d849-00060f2225f1f1e2.journal: Bad message
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b5448cf-00060f287b18ffe2.journal: Bad message
Failed to open journal file ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b5479f9-00060f2b1a9334bf.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b54fd22-00060f3643c94623.journal: Bad message
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b55836c-00060f3fe3e6075d.journal: Bad message
Failed to open journal file ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b55b85a-00060f43a75273db.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b55f6f0-00060f4a61a0aa02.journal: Bad message
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b56a9db-00060f53615ecbf6.journal: Bad message
Failed to open journal file ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b577b97-00060f5ec56ad975.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b577852-00060f5e7f780a22.journal: Bad message
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b57ccc5-00060f61ca701cb7.journal: Bad message
Failed to open journal file ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b585153-00060f680848ec19.journal: Bad message
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b58eee8-00060f7035dc3fa9.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b5913cd-00060f729d4f6a0f.journal: Bad message
Failed to open journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b5b2a72-00060f8f6dedd3f2.journal: Bad message
Failed to open journal file ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b5b4abe-00060f91a0f8c147.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b5bd4bd-00060f9ad8fe2611.journal: Bad message
Failed to open journal file ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b5db5d3-00060fc314acea02.journal: Bad message
Journal effective settings seal=no keyed_hash=yes compress=ZSTD compress_threshold_bytes=8B
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b61040f-00060ffd48fc1634.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b6105f0-00060ffd8f292ab0.journal added.
File ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b61152b-00060fff6e333140.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b623145-00061011673a117b.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b633362-0006101f6ecf8901.journal added.
File ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b6261e0-000610138c0a75d7.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b6356ec-0006102139eaa8b1.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b638a61-00061025af3cc686.journal added.
File ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b63b327-00061027a9e1c9e9.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b64955b-00061039cfde0c35.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b64bc97-0006103d13d24252.journal added.
File ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b64aec0-0006103bc7b92dd2.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b65c2c9-00061052ac2ab208.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b665ecc-0006105f99c33ba5.journal added.
File ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b66966f-000610640367f5d9.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b6738e8-00061071214c5917.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b6949a3-0006109577b57b98.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b69545d-0006109640344716.journal added.
File ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b69e309-000610a05cee0581.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b6ba9c6-000610bc5c617240.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b6e98b4-000610e8f9634f21.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b72d9c4-0006112a9233ddca.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b777769-00061170987ab1e9.journal added.
File ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b789d7f-0006117d68c87a58.journal added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b79546c-000611871627159f.journal added.
File ./system at 000611a3547cb7dc-30ac75b8bb548880.journal~ added.
File ./system at 000611a38b8b4850-ea9184a7bb465911.journal~ added.
File ./system at 564158f161f142e1a9acea4b9e126e5e-000000000b7be30e-000611a354771f2f.journal added.
File ./user-1000 at 000611a38c9f0a95-15a0ef6c330d1d0d.journal~ added.
File ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b7bd2dc-000611a29a8c1866.journal added.
File ./system.journal added.
File ./user-1000.journal added.
File ./user-1001.journal added.
Journal file ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b531a7b-00060f1593b6379e.journal corrupted, ignoring file.
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b61040f-00060ffd48fc1634.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b6105f0-00060ffd8f292ab0.journal
PASS: ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b61152b-00060fff6e333140.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b623145-00061011673a117b.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b633362-0006101f6ecf8901.journal
PASS: ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b6261e0-000610138c0a75d7.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b6356ec-0006102139eaa8b1.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b638a61-00061025af3cc686.journal
PASS: ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b63b327-00061027a9e1c9e9.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b64955b-00061039cfde0c35.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b64bc97-0006103d13d24252.journal
PASS: ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b64aec0-0006103bc7b92dd2.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b65c2c9-00061052ac2ab208.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b665ecc-0006105f99c33ba5.journal
PASS: ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b66966f-000610640367f5d9.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b6738e8-00061071214c5917.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b6949a3-0006109577b57b98.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b69545d-0006109640344716.journal
PASS: ./user-1001 at a10c5e2030f04d2d89a350feb75bc384-000000000b69e309-000610a05cee0581.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b6ba9c6-000610bc5c617240.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b6e98b4-000610e8f9634f21.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b72d9c4-0006112a9233ddca.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b777769-00061170987ab1e9.journal
PASS: ./system at a10c5e2030f04d2d89a350feb75bc384-000000000b789d7f-0006117d68c87a58.journal
PASS: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b79546c-000611871627159f.journal
PASS: ./system at 000611a3547cb7dc-30ac75b8bb548880.journal~
PASS: ./system at 000611a38b8b4850-ea9184a7bb465911.journal~
PASS: ./system at 564158f161f142e1a9acea4b9e126e5e-000000000b7be30e-000611a354771f2f.journal
PASS: ./user-1000 at 000611a38c9f0a95-15a0ef6c330d1d0d.journal~
0015f0: Invalid data hash table hash table item (102560/233016) head_hash_offset: c25c9d8e09f953fc
0015f0: Invalid object contents: Bad message
File corruption detected at ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b7bd2dc-000611a29a8c1866.journal:5616 (of 8388608 bytes, 0%).
FAIL: ./user-1000 at a10c5e2030f04d2d89a350feb75bc384-000000000b7bd2dc-000611a29a8c1866.journal (Bad message)
PASS: ./system.journal
PASS: ./user-1000.journal
PASS: ./user-1001.journal
Root directory . removed.
mmap cache statistics: 1005303406 category cache hit, 44953079 window list hit, 283 miss