[systemd-devel] Changed ordering of systemd-resolved.service

[Dimitri John Ledkov]

On 16 April 2018 at 14:25, Paul Menzel
<pmenzel+systemd-devel at molgen.mpg.de> wrote:
> Dear Dimitri,
>
>
> Thank you for your quick response.
>
>
> On 04/16/18 12:47, Dimitri John Ledkov wrote:
>>
>> On 13 April 2018 at 16:40, Paul Menzel wrote:
>
>
>>> In commit 1f158013 (resolved.service: set DefaultDependencies=no) the
>>> ordering of systemd-resolved.service was changed. (How do I find the
>>> merge
>>> request to find possible discussion? Also the commit message description
>>> is
>>> too specific in my opinion, as it doesn’t give a clue that more is
>>> changed.)
>>
>>
>> https://github.com/systemd/systemd/pull/7609
>
>
> Thank you, no idea, why I didn’t find it with `git log --oneline --graph`.
> Hmm, looks like, Lennart directly put that commit in master without merging
> the pull request.
>
>>> I like starting systemd-resolved earlier, but unfortunately ordering it
>>> before `network.target` adds a delay on systems wanting to start as fast
>>> as
>>> possible. But why did you change it from `network-online.target` to
>>> `network.target`? I’d say `network-online.target` is more correct.
>>>
>>> For my use case of a fast system start-up, this change delays it by at
>>> least
>>> 100 ms, as now it takes longer to reach the end of the network target.
>>
>>
>> cloud-init initializes networking configuration by fetching,
>> potentially, remote sources to customize an instance on first boot.
>> Specifically it may dhcp any interface, to reach a metadata source,
>> download the real networking configuration, reconfigure networking to
>> match the final networking details (all interfaces / public ip
>> addresses / etc), and proceed to complete networking.target and
>> network-online.target.
>>
>> This means that resolved is required earlier in the boot cycle. Before
>> networking.target.
>
>
> Just to be sure, you mean *network.target*, right?
>
> Thank you for specifying the requirement. I agree, that it should be started
> as early as possible, but I disagree with the rest.
>
>> There are things that expect network to be up in
>> "network-online.target", which by some is implied to mean DNS
>> resolution too, unfortunately.
>
>
> Sorry for being ignorant, but could you please be specific, what these
> things are. If these units have that requirement order them after
> `network-online.target`.
>
>>> If your systems have problems with it, they have wrong dependencies,
>>> don’t
>>> they? Also, they should probably be able to deal with the situation, that
>>> DNS does not work, as that can happen during operation.
>>>
>>> So, I’d really like to rework that ordering change.
>>
>>
>> Reworking that change will break certain public cloud providers
>> unfortunately because of public clouds metadata providers being odd.
>>
>> Note, we cannot use dbus activation in this case as dbus-daemon is not
>> up yet, and systemd-resolve command line client also does not work at
>> this point.
>>
>> If you want to make it an optional dependency that early, maybe it
>> will be possible to convert systemd-resolved to be socket activated on
>> tcp/udp?
>>
>> Alternatively, as a system integrator, you may want to change these
>> dependencies in your distro, especially if you do not configure
>> resolved _stub resolver_ as the default provider of /etc/resolv.conf
>> or for example to do not use the recommended default stub-provider
>> over 127.0.0.53 and instead use the nss module over dbus.
>>
>> The above dependencies are correct and recommend, for the default
>> setup of /etc/resolv.conf pointing at the stub-resolv.conf as
>> generated by resolved at runtime.
>>
>> Specifically, the dependencies as is are "too-early" if one uses the
>> last two modes of the /etc/resolv.conf handling as described in the
>> man page -
>> https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html#/etc/resolv.conf
>
>
> First, I think, the terminology of *early* leads to misunderstandings. For
> you it includes ordering with `Before=`, for me it’s just about `After=`
> statements.
>

It's actually both. Cloud-init is a cross-distribution tool, and it
injects itself at multiple points during boot. It pre-empts networking
target, is between networking & network-online, and after
network-online target.

Without this upstream change, cloud-init was not able to pre-empt
network.target, was resulting in a dependency cycle, and systems
resulted booting degraded (due to dependency cycle resolved by
shooting arbitrary unit in the head), in a default upstream systemd
configuration.


> Anyway, regressing the user experience for everyone only because it’s

Can you please explain what has degraded? starting systemd-resolved
before or after network*.target shouldn't make any difference in wall
clock time to reach multi-user.target. And in my boot testing, I did
not see any boot regressions.

Or are you explicitly measuring time to network.target, separate from
time to network-online.target, and separate from reaching the default
target?

Have you been previously booting with network-onlinge.target &
systemd-resolved pulled into the default boot target? And if you were
booting without them, was that expected?

I am also getting multiple support requests for networking and DNS
resolution to be available during emergency and maintenance shell
consoles, thus pulling resolved earlier made a lot of sense to give
root shell at least some ability to talk to the outside world to
download fixes to the system.


> required for cloud-init is not right in my opinion. As you pointed out, the
> system integrator can adapt certain things, and in my opinion, I throw the
> ball back to you, and kindly ask you, to adapt systemd locally so it works
> with your use-case or let’s come up with a better solution.
>

Hm... cloud-init is distribution agnostic, packaged and shipped in
most distributions. And in stock configuration, one would expect any
Linux distro to work nicely with an upstream releases of cloud-init &
systemd.

Please explain the regression you have identified, to design a
solution fit for all purposes.

> Maybe a new target is needed, where you can order your services after, as
> ordering them after systemd-resolved.service is too specific.
>

Possibly, but what are your requirements which you have noticed to
have regressed that we need to fix?

> I submitted a merge/pull request to change the ordering [1].
>

-1 from me.

Please explain, in detail, the regression/bug observed before jumping
onto reverting things. It's not like things are changed without reason
/ without fixing actual production discovered bugs affecting a wide
array of users (due to public cloud nature).

>
> Kind regards,
>
> Paul
>
>
> [1] https://github.com/systemd/systemd/pull/8731
>



-- 
Regards,

Dimitri.