<html>We're seeing problems with some hosts which appears to be caused by the new ad_maximum_machine_account_password_age support in sssd (which notes that a recent adcli is required for this functionality). For some hosts, there is one set of credentials in krb5.keytab, and AD indicates that the machine account password was changed 30 days after the timestamp on those credentials. On other hosts, we see a second set of credentials, 30 days newer than the first set, whose timestamp matches AD's PasswordLastSet property for that host. For the latter set, users report that authentication works some of the time. It may be relevant that this is a large multi-site domain, and we've also found that some hosts have been set up with "ad_site" specified in sssd.conf, pointing at a site that is incorrect (in that it is not the closest site). Both sssd and adcli are out of date on the hosts we're troubleshooting, but I don't see any bugzilla entries, mailing list topics, or release notes that indicate that this is a known problem that has been fixed. Which components should I be looking at when diagnosing this problem further, and what log settings might be most useful as I'm hunting down the problem? Symptoms for host1: System keytab contains old credentials. [root@host1 ~]# klist -kt /etc/krb5.keytab Keytab name: FILE:/etc/krb5.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN 4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN 4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN 4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN 4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN 4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN 4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN 4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN 4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN 4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN 4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN 4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN 4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN 4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN 4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN The date of the new credentials matches the “PasswordLastSet” property on that host in AD: PS H:\> Get-ADComputer -Identity host1 -Properties name,LastLogonDate,PasswordLastSet,modified,modifyTimeStamp DistinguishedName : CN=HOST1,OU=Build,OU=DOMAIN_DevIT,DC=domain,DC=lan DNSHostName : host1.domain.lan Enabled : True LastLogonDate : 6/1/2018 10:41:59 AM Modified : 6/3/2018 3:27:10 PM modifyTimeStamp : 6/3/2018 3:27:10 PM Name : HOST1 ObjectClass : computer ObjectGUID : d21d7f72-xxxx-xxxx-xxxx-aac9cf895d31 PasswordLastSet : 6/3/2018 3:26:59 PM SamAccountName : HOST1$ SID : S-1-5-21-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxx UserPrincipalName : UserPrincipalName : adcli and sssd are out of date on that host: [root@host1 ~]# rpm -q adcli sssd adcli-0.8.1-3.el7.x86_64 sssd-1.15.2-50.el7_4.11.x86_64 The current versions are: $ yum list adcli sssd … Installed Packages sssd.x86_64 1.16.0-19.el7 @base Available Packages adcli.x86_64 0.8.1-4.el7 base Logs include: [sssd[ldap_child[1271]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Preauthentication failed. Unable to create GSSAPI-encrypted LDAP connection. [sssd[ldap_child[1271]]]: Preauthentication failed Symptoms for host2: System keytab contains new credentials, 30 days newer than a previous set. [it@host2 ~]$ sudo klist -kt /etc/krb5.keytab [sudo] password for it: Keytab name: FILE:/etc/krb5.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN 3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN 3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN 3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN 3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN 3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN 3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN 3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN 3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN 3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN 3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN 3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN 3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN 3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN 3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN 4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN 4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN 4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN 4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN 4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN 4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN 4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN 4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN 4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN 4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN 4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN 4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN 4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN 4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN 4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN The date of the new credentials matches the “PasswordLastSet” property on that host in AD: PS H:\> Get-ADComputer -Identity host2 -Properties name,LastLogonDate,PasswordLastSet,modified,modifyTimeStamp DistinguishedName : CN=HOST2,OU=Build,OU=DOMAIN_DevIT,DC=domain,DC=lan DNSHostName : host2.domain.lan Enabled : True LastLogonDate : 7/6/2018 10:51:11 PM Modified : 7/6/2018 10:52:06 PM modifyTimeStamp : 7/6/2018 10:52:06 PM Name : HOST2 ObjectClass : computer ObjectGUID : f70a3bab-xxxx-xxxx-xxxx-932457b18768 PasswordLastSet : 6/21/2018 8:09:02 AM SamAccountName : HOST2$ SID : S-1-5-21-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxx UserPrincipalName : adcli and sssd are out of date on that host: [root@host2 ~]# rpm -q adcli sssd adcli-0.8.1-3.el7.x86_64 sssd-1.14.0-43.el7_3.14.x86_64 The current versions are: $ yum list adcli sssd … Installed Packages sssd.x86_64 1.16.0-19.el7 @base Available Packages adcli.x86_64 0.8.1-4.el7 base Logs include: [sssd[krb5_child[21580]]]: Preauthentication failed </html>