[systemd-devel] Socket activation of container with private network

Mon Apr 20 11:15:55 PDT 2015

Lennart Poettering <lennart at poettering.net> writes:
> On Mon, 20.04.15 13:01, Spencer Baugh (sbaugh at catern.com) wrote:
>> Lennart Poettering <lennart at poettering.net> writes:
>> > Hmm, so you say the initial connection does not work but triggers the
>> > container, but the subsequent one will?
>> 
>> Not quite; the initial connection seems to actually make it to sshd, as
>> sshd has logs of getting it, but the connection is interrupted at some
>> point by some thing before anything useful can be done.
>> Subsequent connections indeed work fine.
>
> Interrupted? What precisely does sshd in the container log about the
> connection?

I've just noticed that there are in fact two cases: The case where I
first ssh from the host to the container, and the case where I first ssh
from another unrelated machine with IPv6 connectivity to the
container. Neither works, but they do appear to have different
behavior. In both cases, all subsequent ssh connections work fine no
matter where they originate from. Here are logs for both cases, both ssh
and sshd side.

Case of sshing from the host to the container:
Both sides are hung at the end of these logs.

# Log of ssh -vvvv on the host
  root at ipv6-test:~# ssh -vvvv 2001:470:8:9d:201:2ff:feaa:bbcd -p 23
  OpenSSH_6.7p1 Debian-3, OpenSSL 1.0.1k 8 Jan 2015
  debug1: Reading configuration data /etc/ssh/ssh_config
  debug1: /etc/ssh/ssh_config line 19: Applying options for *
  debug2: ssh_connect: needpriv 0
  debug1: Connecting to 2001:470:8:9d:201:2ff:feaa:bbcd [2001:470:8:9d:201:2ff:feaa:bbcd] port 23.
  debug1: Connection established.
  debug1: permanently_set_uid: 0/0
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_rsa type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_rsa-cert type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_dsa type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_dsa-cert type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ecdsa type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ecdsa-cert type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ed25519 type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ed25519-cert type -1
  debug1: Enabling compatibility mode for protocol 2.0
  debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-3

# logs of sshd inside the container, when sshing from host
  root at ipv6-container:/# journalctl -u sshd*
  -- Logs begin at Mon 2015-04-20 18:08:32 UTC, end at Mon 2015-04-20 18:08:33 UTC. --
  Apr 20 18:08:32 ipv6-container systemd[1]: Starting SSH Per-Connection Server for 0 ([2001:470:8:9d:201:2ff:feaa:bbcd]:38383)...
  Apr 20 18:08:32 ipv6-container systemd[1]: Started SSH Per-Connection Server for 0 ([2001:470:8:9d:201:2ff:feaa:bbcd]:38383).
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: inetd sockets after dupping: 3, 4
  Apr 20 18:08:32 ipv6-container sshd[57]: Connection from 2001:470:8:9d:201:2ff:feaa:bbcd port 38383 on 2001:470:8:9d:201:2ff:feaa:bbcd port 23
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: Client protocol version 2.0; client software version OpenSSH_6.7p1 Debian-3
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: match: OpenSSH_6.7p1 Debian-3 pat OpenSSH* compat 0x04000000
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: Enabling compatibility mode for protocol 2.0
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
  Apr 20 18:08:32 ipv6-container sshd[57]: debug2: fd 3 setting O_NONBLOCK
  Apr 20 18:08:32 ipv6-container sshd[57]: debug3: fd 4 is O_NONBLOCK
  Apr 20 18:08:32 ipv6-container sshd[57]: debug2: Network child is on pid 64
  Apr 20 18:08:32 ipv6-container sshd[57]: debug3: preauth child monitor started
  Apr 20 18:08:32 ipv6-container sshd[57]: debug3: privsep user:group 104:65534 [preauth]
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: permanently_set_uid: 104/65534 [preauth]
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
  Apr 20 18:08:32 ipv6-container sshd[57]: debug1: SSH2_MSG_KEXINIT sent [preauth]

Case of sshing from an unrelated machine to the container:
The ssh side terminates with the error at the end, but the sshd side
appears to just hang.

# logs of ssh -vvvv on unrelated machine
  root at lxc0:~# ssh -vvvv 2001:470:8:9d:201:2ff:feaa:bbcd -p 23
  OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
  debug1: Reading configuration data /etc/ssh/ssh_config
  debug1: /etc/ssh/ssh_config line 19: Applying options for *
  debug2: ssh_connect: needpriv 0
  debug1: Connecting to 2001:470:8:9d:201:2ff:feaa:bbcd [2001:470:8:9d:201:2ff:feaa:bbcd] port 23.
  debug1: Connection established.
  debug1: permanently_set_uid: 0/0
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_rsa type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_rsa-cert type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_dsa type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_dsa-cert type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ecdsa type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ecdsa-cert type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ed25519 type -1
  debug1: key_load_public: No such file or directory
  debug1: identity file /root/.ssh/id_ed25519-cert type -1
  debug1: Enabling compatibility mode for protocol 2.0
  debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
  debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5
  debug1: match: OpenSSH_6.7p1 Debian-5 pat OpenSSH* compat 0x04000000
  debug2: fd 3 setting O_NONBLOCK
  debug3: put_host_port: [2001:470:8:9d:201:2ff:feaa:bbcd]:23
  debug3: load_hostkeys: loading entries for host "[2001:470:8:9d:201:2ff:feaa:bbcd]:23" from file "/root/.ssh/known_hosts"
  debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:1
  debug3: load_hostkeys: loaded 1 keys
  debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01 at openssh.com,ecdsa-sha2-nistp384-cert-v01 at openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
  debug1: SSH2_MSG_KEXINIT sent
  debug1: SSH2_MSG_KEXINIT received
  debug2: kex_parse_kexinit: curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
  debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01 at openssh.com,ecdsa-sha2-nistp384-cert-v01 at openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01 at openssh.com,ssh-rsa-cert-v01 at openssh.com,ssh-dss-cert-v01 at openssh.com,ssh-rsa-cert-v00 at openssh.com,ssh-dss-cert-v00 at openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se
  debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm at openssh.com,hmac-ripemd160-etm at openssh.com,hmac-sha1-96-etm at openssh.com,hmac-md5-96-etm at openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm at openssh.com,hmac-ripemd160-etm at openssh.com,hmac-sha1-96-etm at openssh.com,hmac-md5-96-etm at openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib
  debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit: first_kex_follows 0
  debug2: kex_parse_kexinit: reserved 0
  debug2: kex_parse_kexinit: curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
  debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at openssh.com
  debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at openssh.com
  debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
  debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
  debug2: kex_parse_kexinit: none,zlib at openssh.com
  debug2: kex_parse_kexinit: none,zlib at openssh.com
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit: first_kex_follows 0
  debug2: kex_parse_kexinit: reserved 0
  debug2: mac_setup: setup umac-64-etm at openssh.com
  debug1: kex: server->client aes128-ctr umac-64-etm at openssh.com none
  debug2: mac_setup: setup umac-64-etm at openssh.com
  debug1: kex: client->server aes128-ctr umac-64-etm at openssh.com none
  debug1: sending SSH2_MSG_KEX_ECDH_INIT
  debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
  Read from socket failed: Connection reset by peer

# logs of sshd inside the container, when sshing from unrelated machine
  -- Logs begin at Mon 2015-04-20 18:06:52 UTC, end at Mon 2015-04-20 18:06:53 UTC. --
  Apr 20 18:06:52 ipv6-container systemd[1]: Starting SSH Per-Connection Server for 0 ([2001:470:7:12f::2]:42531)...
  Apr 20 18:06:52 ipv6-container systemd[1]: Started SSH Per-Connection Server for 0 ([2001:470:7:12f::2]:42531).
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: inetd sockets after dupping: 3, 4
  Apr 20 18:06:52 ipv6-container sshd[57]: Connection from 2001:470:7:12f::2 port 42531 on 2001:470:8:9d:201:2ff:feaa:bbcd port 23
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: Client protocol version 2.0; client software version OpenSSH_6.7p1 Debian-5
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: match: OpenSSH_6.7p1 Debian-5 pat OpenSSH* compat 0x04000000
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: Enabling compatibility mode for protocol 2.0
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
  Apr 20 18:06:52 ipv6-container sshd[57]: debug2: fd 3 setting O_NONBLOCK
  Apr 20 18:06:52 ipv6-container sshd[57]: debug3: fd 4 is O_NONBLOCK
  Apr 20 18:06:52 ipv6-container sshd[57]: debug2: Network child is on pid 67
  Apr 20 18:06:52 ipv6-container sshd[57]: debug3: preauth child monitor started
  Apr 20 18:06:52 ipv6-container sshd[57]: debug3: privsep user:group 104:65534 [preauth]
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: permanently_set_uid: 104/65534 [preauth]
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
  Apr 20 18:06:52 ipv6-container sshd[57]: debug1: SSH2_MSG_KEXINIT sent [preauth]