[systemd-devel] Socket activation of container with private network
sbaugh at catern.com
sbaugh at catern.com
Fri Apr 17 20:27:16 PDT 2015
--text follows this line--
Hi,
I am having trouble with socket-activated containers, where the socket
is first opened outside the container, on an interface/IP address that
is then passed in to the container.
In short, when I try to ssh to the IP address of the container, the
container is indeed activated as it should be, but ssh fails with:
Read from socket failed: Connection reset by peer
I believe this is due to the ssh connection successfully starting then
being interrupted by something unknown before it can prompt for a
password, but not sure what this unknown thing is - systemd, networking
setup, something else?
In more detail, I have a script, interface-setup.sh, to create a
veth. (Contents of the script are at the end of this email.) One end of
the veth is added to a bridge, and the other end gets an IPv6
address. That end will be sent into the container. Outside of the
container, I bind to that address with the following .socket unit.
# /etc/systemd/system/container-nspawn.socket
[Unit]
Description=The SSH socket of my little container
[Socket]
ExecStartPre=/srv/interface-setup.sh
ListenStream=[2001:470:8:9d:201:2ff:feaa:bbcd]:23
ExecStopPost=/srv/interface-teardown.sh
FreeBind=yes
And I have the following corresponding .service unit.
# /etc/systemd/system/container-nspawn.service
[Unit]
Description=Contributed Container for sbaugh/debian-safe
[Service]
ExecStart=/usr/bin/systemd-nspawn --keep-unit -b --network-interface=sbaugh-veth1 --directory=/srv/debian-safe 3
KillMode=process
Inside the container, I have the following two unit files:
# /srv/debian-safe/etc/systemd/system/sshd at .service
[Unit]
Description=SSH Per-Connection Server for %I
[Service]
ExecStart=-/usr/sbin/sshd -i
StandardInput=socket
# /srv/debian-safe/etc/systemd/system/sshd.socket
[Unit]
Description=SSH Socket for Per-Connection Servers
[Socket]
ListenStream=[2001:470:8:9d:201:2ff:feaa:bbcd]:23
# repeat ListenStream twice as per
# http://lists.freedesktop.org/archives/systemd-devel/2015-February/028232.html
ListenStream=[2001:470:8:9d:201:2ff:feaa:bbcd]:23
FreeBind=yes
Accept=yes
I can start the socket on the host just fine:
● container-nspawn.socket - The SSH socket of my little container
Loaded: loaded (/etc/systemd/system/container-nspawn.socket; static; vendor preset: enabled)
Drop-In: /etc/systemd/system/container-nspawn.socket.d
└─override.conf
Active: active (listening) since Fri 2015-04-17 17:21:08 EDT; 17s ago
Listen: [2001:470:8:9d:201:2ff:feaa:bbcd]:23 (Stream)
Process: 1239 ExecStartPre=/srv/interface-setup.sh (code=exited, status=0/SUCCESS)
Apr 17 17:21:08 ipv6-test systemd[1]: Starting The SSH socket of my little container.
Apr 17 17:21:08 ipv6-test systemd[1]: Listening on The SSH socket of my little container.
When I ssh from another machine to this container's IPv6 address, it
gets activated as it should:
● container-nspawn.service - Contributed Container for sbaugh/debian-safe
Loaded: loaded (/etc/systemd/system/container-nspawn.service; static; vendor preset: enabled)
Active: active (running) since Fri 2015-04-17 17:21:31 EDT; 10s ago
Main PID: 1262 (systemd-nspawn)
CGroup: /system.slice/container-nspawn.service
├─1262 /usr/bin/systemd-nspawn --keep-unit -b --network-interface=sbaugh-veth1 --directory=/srv/debian-safe 3
├─1263 /lib/systemd/systemd 3
└─system.slice
├─cron.service
│ └─1332 /usr/sbin/cron -f
├─system-sshd.slice
│ └─sshd at 0-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42162.service
│ ├─1331 sshd: [accepted]
│ └─1343 sshd: [net]
├─systemd-journald.service
│ └─1283 /lib/systemd/systemd-journald
├─systemd-networkd.service
│ └─1278 /lib/systemd/systemd-networkd
├─console-getty.service
│ └─1339 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt102
└─rsyslog.service
└─1334 /usr/sbin/rsyslogd -n
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Started Permit User Sessions.
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: Starting Console Getty...
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Started Console Getty.
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Reached target Login Prompts.
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Started System Logging Service.
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Reached target Multi-User System.
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: Starting Update UTMP about System Runlevel Changes...
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Started Cleanup of Temporary Directories.
Apr 17 17:21:32 ipv6-test systemd-nspawn[1262]: [ OK ] Started Update UTMP about System Runlevel Changes.
Apr 17 17:21:33 ipv6-test systemd-nspawn[1262]: Debian GNU/Linux 8 ipv6-test console
However, the ssh fails with, as I said above:
Read from socket failed: Connection reset by peer
>From running ssh -vvvv (output attached), and from the server-side logs,
it seems that the connection does start, but is cut off at some point.
Inside the container, after sshing once and failing with the above
error:
root at ipv6-test:/# systemctl status sshd.socket
● sshd.socket - SSH Socket for Per-Connection Servers
Loaded: loaded (/etc/systemd/system/sshd.socket; enabled)
Active: active (listening) since Fri 2015-04-17 21:30:27 UTC; 11s ago
Listen: [2001:470:8:9d:201:2ff:feaa:bbcd]:23 (Stream)
[2001:470:8:9d:201:2ff:feaa:bbcd]:23 (Stream)
Accepted: 1; Connected: 1
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root at ipv6-test:/# systemctl status sshd@*
● sshd at 0-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42181.service - SSH Per-Connection Server for 0 ([2001:470:7:12f::2]:42181)
Loaded: loaded (/etc/systemd/system/sshd at .service; static)
Active: active (running) since Fri 2015-04-17 21:42:42 UTC; 5s ago
Main PID: 57 (sshd)
CGroup: /system.slice/container-nspawn.service/system.slice/system-sshd.slice/sshd at 0-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42181.service
├─57 sshd: [accepted]
└─68 sshd: [net]
Apr 17 21:42:42 ipv6-test sshd[57]: Connection from 2001:470:7:12f::2 port 42181 on 2001:470:8:9d:201:2ff:feaa:bbcd port 23
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: Client protocol version 2.0; client software version OpenSSH_6.7p1 Debian-5
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: match: OpenSSH_6.7p1 Debian-5 pat OpenSSH* compat 0x04000000
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: Enabling compatibility mode for protocol 2.0
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
Apr 17 21:42:43 ipv6-test sshd[57]: debug2: fd 3 setting O_NONBLOCK
Apr 17 21:42:43 ipv6-test sshd[57]: debug2: Network child is on pid 68
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: permanently_set_uid: 104/65534 [preauth]
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: SSH2_MSG_KEXINIT sent [preauth]
If another ssh connection is attempted, it goes through just
fine. Inside the container after sshing again:
root at ipv6-test:/# systemctl status sshd.socket
● sshd.socket - SSH Socket for Per-Connection Servers
Loaded: loaded (/etc/systemd/system/sshd.socket; enabled)
Active: active (listening) since Fri 2015-04-17 21:42:42 UTC; 2min 27s ago
Listen: [2001:470:8:9d:201:2ff:feaa:bbcd]:23 (Stream)
[2001:470:8:9d:201:2ff:feaa:bbcd]:23 (Stream)
Accepted: 2; Connected: 1
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root at ipv6-test:/# systemctl status sshd@*
● sshd at 0-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42181.service - SSH Per-Connection Server for 0 ([2001:470:7:12f::2]:42181)
Loaded: loaded (/etc/systemd/system/sshd at .service; static)
Active: active (running) since Fri 2015-04-17 21:42:42 UTC; 1min 52s ago
Main PID: 57 (sshd)
CGroup: /system.slice/container-nspawn.service/system.slice/system-sshd.slice/sshd at 0-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42181.service
├─57 sshd: [accepted]
└─68 sshd: [net]
Apr 17 21:42:42 ipv6-test sshd[57]: Connection from 2001:470:7:12f::2 port 42181 on 2001:470:8:9d:201:2ff:feaa:bbcd port 23
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: Client protocol version 2.0; client software version OpenSSH_6.7p1 Debian-5
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: match: OpenSSH_6.7p1 Debian-5 pat OpenSSH* compat 0x04000000
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: Enabling compatibility mode for protocol 2.0
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
Apr 17 21:42:43 ipv6-test sshd[57]: debug2: fd 3 setting O_NONBLOCK
Apr 17 21:42:43 ipv6-test sshd[57]: debug2: Network child is on pid 68
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: permanently_set_uid: 104/65534 [preauth]
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Apr 17 21:42:43 ipv6-test sshd[57]: debug1: SSH2_MSG_KEXINIT sent [preauth]
● sshd at 1-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42182.service - SSH Per-Connection Server for 1 ([2001:470:7:12f::2]:42182)
Loaded: loaded (/etc/systemd/system/sshd at .service; static)
Active: active (running) since Fri 2015-04-17 21:44:33 UTC; 1s ago
Main PID: 73 (sshd)
CGroup: /system.slice/container-nspawn.service/system.slice/system-sshd.slice/sshd at 1-2001:470:8:9d:201:2ff:feaa:bbcd:23-2001:470:7:12f::2:42182.service
├─73 sshd: root [priv]
└─74 sshd: root [net]
Apr 17 21:44:33 ipv6-test sshd[73]: debug1: attempt 0 failures 0 [preauth]
Apr 17 21:44:33 ipv6-test sshd[73]: debug2: parse_server_config: config reprocess config len 717
Apr 17 21:44:33 ipv6-test sshd[73]: debug2: monitor_read: 8 used once, disabling now
Apr 17 21:44:33 ipv6-test sshd[73]: debug2: input_userauth_request: setting up authctxt for root [preauth]
Apr 17 21:44:33 ipv6-test sshd[73]: debug1: PAM: initializing for "root"
Apr 17 21:44:33 ipv6-test sshd[73]: debug1: PAM: setting PAM_RHOST to "2001:470:7:12f::2"
Apr 17 21:44:33 ipv6-test sshd[73]: debug1: PAM: setting PAM_TTY to "ssh"
Apr 17 21:44:33 ipv6-test sshd[73]: debug2: monitor_read: 100 used once, disabling now
Apr 17 21:44:33 ipv6-test sshd[73]: debug2: input_userauth_request: try method none [preauth]
Apr 17 21:44:33 ipv6-test sshd[73]: debug2: monitor_read: 4 used once, disabling now
So it seems that the socket activation is successful, and the ssh
connection goes through on the socket bound outside the container, but
at some point the connection is severed - by what?
Thank you for any help.
Additional information:
# interface-setup.sh
#!/bin/bash
name=sbaugh
set -o errexit
set -o nounset
ip link add \
name $name-veth0 \
addr 00:01:02:aa:bb:cc \
type veth \
peer \
name $name-veth1 \
addr 00:01:02:aa:bb:cd
echo 2 > /proc/sys/net/ipv6/conf/$name-veth0/accept_ra
echo 2 > /proc/sys/net/ipv6/conf/$name-veth1/accept_ra
brctl addif br0 $name-veth0
ip link set $name-veth0 up
ip link set $name-veth1 up
# interface-teardown.sh
#!/bin/bash
ip link delete sbaugh-veth0
# ip addr, after starting the container
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:00:80:ed:9d:08 brd ff:ff:ff:ff:ff:ff
inet 128.237.157.8/24 brd 128.237.157.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::200:80ff:feed:9d08/64 scope link
valid_lft forever preferred_lft forever
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 00:01:02:aa:bb:cc brd ff:ff:ff:ff:ff:ff
inet6 2001:470:8:9d::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::21:5cff:fe68:1709/64 scope link
valid_lft forever preferred_lft forever
4: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default
link/sit 0.0.0.0 brd 0.0.0.0
5: he-ipv6 at NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default
link/sit 128.237.157.8 peer 216.66.22.2
inet6 2001:470:7:9d::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::80ed:9d08/64 scope link
valid_lft forever preferred_lft forever
23: sbaugh-veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether 00:01:02:aa:bb:cc brd ff:ff:ff:ff:ff:ff
inet6 fe80::201:2ff:feaa:bbcc/64 scope link
valid_lft forever preferred_lft forever
<#part type="text/plain" filename="~/sshlog" disposition=attachment description="ssh client-side log, fail case">
<#/part>
More information about the systemd-devel
mailing list