[systemd-devel] Antw: how to debug kernel panic which generated by udevadm at systemd?

Tue Oct 15 12:36:41 UTC 2019

Hi!

I think your driver loads OK, but it is not working correctly. Maybe enable
kernel debug messages and add debug prints to your driver, especially regarding
any queries. I suspect some NULL pointer or empty string being returned.

Regards,
Ulrich

>>> www <ouyangxuan10 at 163.com> schrieb am 15.10.2019 um 13:46 in Nachricht
<252176e7.9975.16dcf3de51e.Coremail.ouyangxuan10 at 163.com>:
> Dear all,
> 
> 
> I add a new driver to kernel, and it probe success. When enter into systemd,

> the udevadm generate a kernel panic.
> I want to ask how to debug it and find out where the error occurred? When 
> did udevadm load? What commands are used by udevadm, and what are the 
> specific operations?
> 
> 
> [    3.291819] pmbus 13-0058: PMBus status register not found
> [    3.297940] pmbus 13-0059: PMBus status register not found
> [    3.305830] usbcore: registered new interface driver usbhid
> [    3.311417] usbhid: USB HID core driver
> [    3.326527] aspeed_jtag: driver successfully loaded.            -->      

> load the driver success
> [    3.335427] NET: Registered protocol family 10
> [    3.345639] Segment Routing with IPv6
> [    3.349995] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
> [    3.357507] NET: Registered protocol family 17
> [    3.362012] 8021q: 802.1Q VLAN Support v1.8
> [    3.379650] printk: console [netcon0] enabled
> [    3.384027] netconsole: network logging started
> [    3.390140] input: gpio-keys as /devices/platform/gpio-keys/input/input0
> [    3.398913] hctosys: unable to open rtc device (rtc0)
> [    3.411203] Freeing unused kernel memory: 1024K
> [    3.419068] Checked W+X mappings: passed, no W+X pages found
> [    3.424841] Run /init as init process
> rofs = mtd4 squashfs rwfs = mtd5 jffs2
> [    4.392201] jffs2: notice: (681) jffs2_build_xattr_subsystem: complete 
> building xattr subsystem, 8 of xdatum (0 unchecked, 1 orphan) and 11 of xref

> (1 dead, 0 orphan) found.
> [    4.448757] overlayfs: upper fs does not support tmpfile.
> [    6.760748] systemd[1]: System time before build time, advancing clock.
> [    6.893492] systemd[1]: systemd 242-19-gdb2e367+ running in system mode.

> (+PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK +SYSVINIT -UTMP -LIBCRYPTSETUP 
> -GCRYPT -GNUTLS -ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS -KMOD -IDN2 -IDN 
> -PCRE2 default-hierarchy=hybrid)
> [    6.917878] systemd[1]: Detected architecture arm.
> 
> 
> Welcome to Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) 
> 2.7.0-dev!
> 
> 
> [    7.033813] systemd[1]: Set hostname to <demoboard>.
> [    7.769096] systemd[1]: File 
> /lib/systemd/system/systemd-journald.service:12 configures an IP firewall 
> (IPAddressDeny=any), but the local system does not support BPF/cgroup based

> firewalling.
> [    7.786341] systemd[1]: Proceeding WITHOUT firewalling in effect! (This 
> warning is only shown for the first loaded unit using IP firewalling.)
> [    8.942114] systemd[1]: /lib/systemd/system/phosphor-ipmi-net at .socket:3:

> Invalid interface name, ignoring: sys-subsystem-net-devices-%i.device
> [    8.982380] systemd[1]: /lib/systemd/system/dbus.socket:4: ListenStream=

> references a path below legacy directory /var/run/, updating 
> /var/run/dbus/system_bus_socket â†’ /run/dbus/system_bus_socket; please 
> update the unit file accordingly.
> [    9.111166] systemd[1]: Reached target Fans Ready.
> [  OK  ] Reached target Fans Ready.
> [    9.148717] systemd[1]: Created slice 
> system-phosphor\x2dipmi\x2dkcs.slice.
> [  OK  ] Created slice system-phosphor\x2dipmi\x2dkcs.slice.
> [    9.199247] systemd[1]: Listening on udev Kernel Socket.
> [  OK  ] Listening on udev Kernel Socket.
> [  OK  ] Reached target Slices.
> [  OK  ] Created slice system-phosphor\x2dipmi\x2dnet.slice.
> [  OK  ] Reached target Remote File Systems.
> [  OK  ] Created slice system-mapper\x2dwait.slice.
> [  OK  ] Started Forward Password Râ€|uests to Wall Directory Watch.
> [  OK  ] Created slice system-phosphor\x2dgpio\x2dmonitor.slice.
> [  OK  ] Listening on Journal Socket.
>          Mounting Kernel Configuration File System...
> [  OK  ] Started Dispatch Password â€|ts to Console Directory Watch.
> [  OK  ] Reached target Paths.
> [  OK  ] Created slice system-phospâ€|dcertificate\x2dmanager.slice.
> [  OK  ] Listening on initctl Compatibility Named Pipe.
>          Mounting Kernel Debug File System...
> [  OK  ] Created slice system-obmc\x2dled\x2dgroup\x2dstart.slice.
> [  OK  ] Reached target Swap.
>          Mounting Temporary Directory (/tmp)...
> [  OK  ] Listening on Network Service Netlink Socket.
> [  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
> [  OK  ] Created slice system-getty.slice.
> [  OK  ] Listening on udev Control Socket.
>          Starting udev Coldplug all Devices...
> [  OK  ] Listening on Syslog Socket.
> [  OK  ] Created slice system-obmc\x2dconsole.slice.
> [  OK  ] Created slice system-serial\x2dgetty.slice.
> [  OK  ] Created slice system-obmc\x2dread\x2deeprom.slice.
>          Starting Apply Kernel Variables...
>          Starting Remount Root and Kernel File Systems...
> [  OK  ] Started Set initial fan speed when BMC starts..
> [  OK  ] Listening on Journal Socket (/dev/log).
>          Starting Journal Service...
> [  OK  ] Mounted Kernel Configuration File System.
> [  OK  ] Mounted Kernel Debug File System.
> [  OK  ] Mounted Temporary Directory (/tmp).
> [  OK  ] Started Remount Root and Kernel File Systems.
>          Starting Rebuild Hardware Database...
>          Starting Create System Users...
> [  OK  ] Started Apply Kernel Variables.
> [  OK  ] Started Rebuild Hardware Database.
> [  OK  ] Started Create System Users.
> [  OK  ] Started Journal Service.
>          Starting Flush Journal to Persistent Storage...
>          Starting Create Static Device Nodes in /dev...
> [  OK  ] Started Create Static Device Nodes in /dev.
> [   12.129145] systemd-journald[710]: Received request to flush runtime 
> journal from PID 1
>          Starting udev Kernel Device Manager...
> [  OK  ] Reached target Local File Systems (Pre).
>          Mounting /var/volatile...
> [  OK  ] Started Flush Journal to Persistent Storage.
> [  OK  ] Mounted /var/volatile.
>          Starting Load/Save Random Seed...
> [  OK  ] Reached target Local File Systems.
>          Starting Create Volatile Files and Directories...
> [  OK  ] Started Load/Save Random Seed.
> [   13.497500] Unable to handle kernel paging request at virtual address 
> 0179a7b0
> [   13.504866] pgd = 6cf98158
> [   13.507598] [0179a7b0] *pgd=00000000
> [   13.511199] Internal error: Oops: 5 [#1] ARM
> [   13.515489] CPU: 0 PID: 846 Comm: udevadm Not tainted 
> 5.1.5-yocto-s-dirty-e03b6d3 #1              -->     udevadm lead to kernel 
> panic
> [   13.523227] Hardware name: Generic DT based system
> [   13.528031] PC is at strlen+0xc/0x38
> [   13.531624] LR is at kstrdup+0x28/0x60
> [   13.535369] pc : [<806f08ac>]    lr : [<8020dc7c>]    psr: 20000013
> [   13.541625] sp : 999edcf0  ip : 999edd00  fp : 999edcfc
> [   13.546842] r10: 00001000  r9 : 9e753d80  r8 : 80a07008
> [   13.552061] r7 : 999edd5e  r6 : 0179a7b0  r5 : 804004f4  r4 : 00000cc0
> [   13.558579] r3 : 9ac737c0  r2 : 00000000  r1 : 00000cc0  r0 : 0179a7b0
> [   13.565097] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment

> none
> [   13.572220] Control: 00c5387d  Table: 999f8008  DAC: 00000051
> [   13.577964] Process udevadm (pid: 846, stack limit = 0x1ca521ae)
> [   13.583965] Stack: (0x999edcf0 to 0x999ee000)
> [   13.588320] dce0:                                     999edd1c 999edd00 
> 8020dc7c 806f08ac
> [   13.596495] dd00: 9ac47608 9ac47600 999edd60 999edd5e 999edd2c 999edd20 
> 804004f4 8020dc60
> [   13.604668] dd20: 999edd4c 999edd30 804475c0 804004c0 9ac47608 9adb5000 
> 9ac47600 9ac47608
> [   13.612839] dd40: 999edd94 999edd50 804477bc 8044754c 999edd60 999edd60 
> 00000894 0000ff1c
> [   13.621013] dd60: 00000000 00000000 00000000 b7accb6a 9ac47608 9e487940 
> 9ac47608 9adb5000
> [   13.629186] dd80: 9ac47608 99965000 999eddbc 999edd98 80445d54 80447630 
> 9accee10 80a38ba4
> [   13.637356] dda0: 99965000 9ac47608 00000fff 9e753d80 999eddd4 999eddc0 
> 80445ba4 80445cb8
> [   13.645531] ddc0: 9accee10 8073f7e4 999ede04 999eddd8 802acd08 80445b84 
> 802acc78 9accee10
> [   13.653702] dde0: 9990ba40 00400cc0 00000000 999edf60 9accee28 00000001 
> 999ede14 999ede08
> [   13.661876] de00: 802ab5f4 802acc84 999ede5c 999ede18 8025efb0 802ab5cc 
> 9e6fc000 999edf60
> [   13.670048] de20: 9990ba40 9accee40 01b071f8 00001000 7eb787f8 9e753d80 
> 00001000 802ac0a0
> [   13.678221] de40: 999edf60 999edf60 999edf60 00000000 999ede9c 999ede60 
> 802ac0e0 8025ee60
> [   13.686393] de60: 999edefc 999ede70 8023f368 01b071f8 0000000d 9990ba40 
> 80a07008 802ac0a0
> [   13.694566] de80: 999edf60 00001000 999edf60 00000000 999edf24 999edea0 
> 80239348 802ac0ac
> [   13.702737] dea0: 00001000 00000000 00001000 00000000 00000000 00000000 
> 5da3e857 2b82ea81
> [   13.710911] dec0: 5da3e857 2b82ea81 5da3e857 2b82ea81 00001d14 00000000 
> 80a07008 b7accb6a
> [   13.719084] dee0: 01b071f8 80a07008 7eb787f8 802394fc 999edfa4 b7accb6a 
> 8023fc40 00001000
> [   13.727255] df00: 9990ba40 01b071f8 999edf60 00000001 00000000 00000000 
> 999edf54 999edf28
> [   13.735429] df20: 80239520 80239310 0000000d 8025930c 9990ba40 9990ba40 
> 80a07008 01b071f8
> [   13.743600] df40: 00000000 00000000 999edf94 999edf58 8023994c 80239498 
> 00000000 00000000
> [   13.751775] df60: 00000000 00000000 2b82ea81 b7accb6a 01ad4150 01b071f8 
> 457f1654 00000003
> [   13.759947] df80: 801011e4 999ec000 999edfa4 999edf98 802399dc 802398e8 
> 00000000 999edfa8
> [   13.768118] dfa0: 80101000 802399d0 01ad4150 01b071f8 00000005 01b071f8 
> 00001000 00000000
> [   13.776292] dfc0: 01ad4150 01b071f8 457f1654 00000003 00001001 457f11bc 
> 00001001 000005e8
> [   13.784463] dfe0: 00000070 7eb78868 4571bad4 45774f28 60000010 00000005 
> 00000000 00000000
> [   13.792626] Backtrace:
> [   13.795094] [<806f08a0>] (strlen) from [<8020dc7c>] (kstrdup+0x28/0x60)
> [   13.801731] [<8020dc54>] (kstrdup) from [<804004f4>]
(misc_devnode+0x40/0x44)
> [   13.808869]  r7:999edd5e r6:999edd60 r5:9ac47600 r4:9ac47608
> [   13.814546] [<804004b4>] (misc_devnode) from [<804475c0>] 
> (device_get_devnode+0x80/0xe4)
> [   13.822639] [<80447540>] (device_get_devnode) from [<804477bc>] 
> (dev_uevent+0x198/0x2cc)
> [   13.830724]  r7:9ac47608 r6:9ac47600 r5:9adb5000 r4:9ac47608
> [   13.836390] [<80447624>] (dev_uevent) from [<80445d54>] 
> (uevent_show+0xa8/0x124)
> [   13.843788]  r8:99965000 r7:9ac47608 r6:9adb5000 r5:9ac47608 r4:9e487940
> [   13.850491] [<80445cac>] (uevent_show) from [<80445ba4>] 
> (dev_attr_show+0x2c/0x58)
> [   13.858064]  r9:9e753d80 r8:00000fff r7:9ac47608 r6:99965000 r5:80a38ba4

> r4:9accee10
> [   13.865825] [<80445b78>] (dev_attr_show) from [<802acd08>] 
> (sysfs_kf_seq_show+0x90/0x108)
> [   13.873994]  r5:8073f7e4 r4:9accee10
> [   13.877581] [<802acc78>] (sysfs_kf_seq_show) from [<802ab5f4>] 
> (kernfs_seq_show+0x34/0x38)
> [   13.885839]  r10:00000001 r9:9accee28 r8:999edf60 r7:00000000 r6:00400cc0

> r5:9990ba40
> [   13.893658]  r4:9accee10 r3:802acc78
> [   13.897250] [<802ab5c0>] (kernfs_seq_show) from [<8025efb0>] 
> (seq_read+0x15c/0x4ec)
> [   13.904913] [<8025ee54>] (seq_read) from [<802ac0e0>] 
> (kernfs_fop_read+0x40/0x1c4)
> [   13.912481]  r10:00000000 r9:999edf60 r8:999edf60 r7:999edf60 r6:802ac0a0

> r5:00001000
> [   13.920296]  r4:9e753d80
> [   13.922850] [<802ac0a0>] (kernfs_fop_read) from [<80239348>] 
> (__vfs_read+0x44/0x188)
> [   13.930595]  r10:00000000 r9:999edf60 r8:00001000 r7:999edf60 r6:802ac0a0

> r5:80a07008
> [   13.938411]  r4:9990ba40
> [   13.940952] [<80239304>] (__vfs_read) from [<80239520>]
(vfs_read+0x94/0x120)
> [   13.948082]  r10:00000000 r9:00000000 r8:00000001 r7:999edf60 r6:01b071f8

> r5:9990ba40
> [   13.955900]  r4:00001000
> [   13.958442] [<8023948c>] (vfs_read) from [<8023994c>]
(ksys_read+0x70/0xe8)
> [   13.965399]  r9:00000000 r8:00000000 r7:01b071f8 r6:80a07008 r5:9990ba40

> r4:9990ba40
> [   13.973141] [<802398dc>] (ksys_read) from [<802399dc>]
(sys_read+0x18/0x1c)
> [   13.980099]  r9:999ec000 r8:801011e4 r7:00000003 r6:457f1654 r5:01b071f8

> r4:01ad4150
> [   13.987841] [<802399c4>] (sys_read) from [<80101000>] 
> (ret_fast_syscall+0x0/0x54)
> [   13.995319] Exception stack(0x999edfa8 to 0x999edff0)
> [   14.000368] dfa0:                   01ad4150 01b071f8 00000005 01b071f8 
> 00001000 00000000
> [   14.008542] dfc0: 01ad4150 01b071f8 457f1654 00000003 00001001 457f11bc 
> 00001001 000005e8
> [   14.016710] dfe0: 00000070 7eb78868 4571bad4 45774f28
> [   14.021762] Code: e89da800 e1a0c00d e92dd800 e24cb004 (e5d03000)
> [   14.028037] ---[ end trace 850b9c31df4945cf ]---
> [   14.032675] Kernel panic - not syncing: Fatal exception
> [   14.037912] ---[ end Kernel panic - not syncing: Fatal exception ]---
> 
> 
> 
> 
> thanks，
> Byron