All messages come from deployment-mediawiki04 which is a new MW App we have added yesterday and runs Jessie (vs Trusty). Might be related

[8:38:39] <dcausse> yes curl error codes are not very explicit: https://curl.haxx.se/libcurl/c/libcurl-errors.html
[8:38:47] <dcausse> 60 seems to be CURLE_SSL_CACERT

In T142558#2544603, @dcausse wrote:

Tweaks needed to be done:

curl settings appear to be broken and https is failing.
The trick was to download the cert file from relforge1001:/usr/local/share/ca-certificates/Puppet_Internal_CA.crt
copy this file to lxc-vagrant box
make sure that curl will load this file
(identify the cert file opened by default strace curl https://relforge1001.eqiad.wmnet:9243 2>&1 | grep stat)
symlink /etc/ssl/certs/c5aaad6f.0 to this file
It's unclear why curl refused to use the bundle /etc/ssl/certs/ca-certificates.crt ...

Copy pasting from IRC investigations with dcausse / gehel / elukey. To repro:

root@deployment-mediawiki04: strace -e stat -f curl https://deployment-elastic06.deployment-prep.eqiad.wmflabs:9243
stat("/etc/ssl/certs/7abfb60b.0", 0x7ffe7ec372a0) = -1 ENOENT (No such file or directory)
curl: (60) SSL certificate problem: unable to get local issuer certificate

Hashes are different though:

ls -l /etc/ssl/certs/ | grep -i puppet
[11:01:40]  <elukey>	lrwxrwxrwx 1 root root     22 Sep 12 13:50 a861a8e4.0 -> Puppet_Internal_CA.pem
[11:01:43]  <elukey>	lrwxrwxrwx 1 root root     22 Sep 12 13:50 c934af94.0 -> Puppet_Internal_CA.pem
[11:01:45]  <elukey>	lrwxrwxrwx 1 root root     55 Sep 12 13:50 Puppet_Internal_CA.pem -> /usr/local/share/ca-certificates/Puppet_Internal_CA.crt

From the syslog on deployment-mediawiki04

update-ca-certificate is run, then the Puppet_Internal_CA.crt is updated since the beta cluster uses its own puppet master. The subject hash change but the symlinks are not updated:

First puppet run:

Sep 12 13:50:58 deployment-mediawiki04 puppet-agent[1436]: (/Stage[main]/Sslcert/Exec[update-ca-certificates]) Triggered 'refresh' from 6 events

The Puppet_Internal_CA file is then later changed:

Sep 12 14:03:17 (/Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]
  /File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt])
  Filebucketed /usr/local/share/ca-certificates/Puppet_Internal_CA.crt to puppet
  with sum 9f3978d4816ae16ad737cf46ca10af19

Sep 12 14:03:17 (/Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]
  /File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content) content changed
  '{md5}9f3978d4816ae16ad737cf46ca10af19' to
  '{md5}05f38ae56b7395fa123c2b14658702c9'

Sep 12 14:03:17 (/Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]
  /File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt])
  Scheduling refresh of Exec[update-ca-certificates]

Sep 12 14:03:34 (/Stage[main]/Sslcert/Exec[update-ca-certificates]) Triggered 'refresh' from 1 events

Seems update-ca-certificatesfails to find that the .crt file has been changed (and hence the subject hash) and does not update the symlink.

I have saved the syslog files on deployment-mediawiki04 in /root/syslog.T145609.

Seems to me that whenever the .crt file is modified by in sslcert:ca, that should rebuild the symlinks with update-ca-certificates --fresh.

Removing the Search tag from this ticket - please let us know if there is anything else for us to investigate.

Mentioned in SAL (#wikimedia-releng) [2016-10-14T11:30:05Z] <dcausse> deployment-prep running sudo update-ca-certificates --fresh on deployment-ton to fix curl error code 60 in cirrus maint script (T145609)

Change 315934 had a related patch set uploaded (by Gehel):
ssl - make sure certificates are updated "fresh"

https://gerrit.wikimedia.org/r/315934

Change 315934 abandoned by Gehel:
ssl - make sure certificates are updated "fresh"

Reason:
I knew I was missing something. Thanks Faidon.

I'm dropping this, I have no idea how to improve on it.

https://gerrit.wikimedia.org/r/315934

As pointed by @faidon, when running update-ca-certificates with -f there is a small window of time where the certificates symlinks disappear before being recreated. Ideally change of certificates should be done with a different file name, but this is not always practical.