[systemd-devel] Spurious failures starting ConnMan with systemd

[Daniel Wagner]

[Cc: systemd mailing list"

what happened until now
"""
I am working on a system based on Yocto Project which uses ConnMan
v1.33 for network management. The system uses systemd and generally
works well so far, but once in a blue moon ConnMan fails to start up.
Here is the relevant log from systemd:

[   15.886270] systemd[1]: connman.service: Main process exited, 
code=exited, status=1/FAILURE
[   15.912838] systemd[1]: Failed to start Connection service.

There were no logs from connmand in this case, so in the
connman.service file I've added the -d option to ExecStart= and set
StandardOutput to syslog. I have restarted the system many times until
the failure occurred again. There is only a single line from connmand
in the log:

[   12.174528] connmand[278]: Failed to connect to socket 
/var/run/dbus/system_bus_socket: No such file or directory
[   12.209448] systemd[1]: connman.service: Main process exited, 
code=exited, status=1/FAILURE
[   12.242872] systemd[1]: Failed to start Connection service.

So the system D-Bus is not up and running the time ConnMan is started.
In case of failure, I've noticed there is also no mentioning about
D-Bus starting up in the log, but in all other logs there is.

Now, I think the After=dbus.service line in src/connman.service.in is
not enough because After= specifies order, but not a requirement
dependency. This what Requires= is there for. I've added the line

     Requires=dbus.service

to the service file, reverted my other changes, and tried again. I've
been unable to reproduce the problem ever since, so for me the problem
seems to be fixed.

What do you think? Should src/connman.service.in be changed to include
the Requires= line by default or is this something to be fixed
somewhere else?

Should network-pre.target and systemd-sysusers.service be added to
Requires= as well?
"""
]

On 11/15/2016 10:45 AM, Robert Tiemann wrote:
> On 11/14/2016 01:30 PM, Patrik Flykt wrote:
>> On Fri, 2016-11-11 at 12:04 +0100, Daniel Wagner wrote:
>>>
>>> I am no systemd expert. I just did a quick read in the documentation
>>> and
>>> it seems it is recommended to have After= and Requires= together:
>
> Yes, this seems to be required here.
>
>>> Care to send a patch which adds Requires=? Please include your
>>> excellent error report into the commit message.
>
> I can create a patch that adds Requires=dbus.service to the service
> file if you like, but please read my comments below.
>
>> Hmm, then again man systemd.service says that "...Service units with
>> this option configured implicitly gain dependencies on the dbus.socket
>> unit. This type is the default if BusName= is specified..."
>>
>> ConnMan defines both Type=dbus and Busname=net.connman, so it should
>> already have these dependencies. Unless the DefaultDependencies=false
>> overrides this behavior. Do we have any systemd experts around?
>
> I am by no means a systemd expert, so instead I've tried a few things.
> The output of
>
>     $ systemctl show connman.service
>
> with the service file from v1.33 installed does not show any
> dependencies on dbus.socket, but only an After= dependency on
> dbus.service. The only Requires= dependency is on -.mount (and then
> there is also the RequiredMountsFor= dependency). I always thought
> this should be enough because dbus.service is actually required by
> other services in the system, therefore ConnMan should be started
> *after* it, right?
>
> Well. As I've come to realize, the troubles started after an upgrade
> of ConnMan v1.30 to v1.33 a few weeks ago. So, I've checked the
> changes of connman.service.in.
>
> With the service file shipped with v1.30, the output of the above
> command shows Requires=basic.target, which depends on sockets.target,
> which depends on dbus.socket. I think that's why I didn't have any
> trouble with v1.30.
>
> In 09aa024 (first change to the service file after v1.30 release),
> DefaultDependencies was set to false. According to
> https://www.freedesktop.org/software/systemd/man/systemd.special.html,
> an After= dependency on basic.target is added automatically if
> DefaultDependencies is set to yes, so this is missing now. The After=
> dependency on dbus.service is still there, but it is not Required. Out
> of curiosity, I've removed the dbus.service dependency completely to
> check if Type=dbus is enough as is documented. Turns out it is not.
> Neither the output of "systemctl show connman.service" nor that of
> "systemctl list-dependencies connman.service" show _any_ dependencies
> on dbus.service/dbus.socket in this case.
>
> Why the automatically added After= dependency on basic.target has
> turned into a Requires= dependency with the v1.30 file, but the
> explicit After=dbus.service dependency in the file from 09aa024
> remains an After= dependency is beyond me. My guess is that
> DefaultDependencies does a little bit more than is documented, or
> maybe it's just a bug in my version of systemd (v225). Maybe it's some
> strange interaction between unit files that I am unable to see.
>
> A comment from someone with a little more experience with systemd
> would be most welcome.

Any recommendation from systemd folks?

cheers,
daniel