The script source doesn't say so, but I've noticed that it's written by ripienaar. Latest upstream implements this feature:

Yeah I haven't seen recurrence of this so I'm closing the ticket. Thanks.

TLDR: the rough estimate is about 32Mbit/sec from jobrunners to elasticsearch nodes. Traffic is bursty so I advise planning for a 50-60Mbit ceiling.

Regarding running puppetmaster on !Precise: when I tried with Trusty I got this:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep

It seems to be due to a bad interaction between Rails and Activerecord (deb: ruby-activerecord-3.2), this ticket proposes some workarounds:
https://projects.puppetlabs.com/issues/9290

logstash1001-1003: These hosts are older than 1004-1006, and run Precise instead of Jessie. Gmond wouldn't stop or start.

gage@logstash1002:~$ sudo /usr/sbin/gmond -f
[apache_status] Received the following parameters
{'url': 'http://127.0.0.1:80/server-status', 'collect_ssl': 'False', 'metric_group': 'apache'}
Fatal Python error: PyThreadState_Get: no current thread

Jul  9 00:06:37 logstash1002 kernel: [13984018.998725] init: ganglia-monitor main process (14196) killed by ABRT signal
Jul  9 00:06:37 logstash1002 kernel: [13984018.998751] init: ganglia-monitor main process ended, respawning
Jul  9 00:06:37 logstash1002 kernel: [13984019.086671] init: ganglia-monitor main process (14210) killed by ABRT signal
Jul  9 00:06:37 logstash1002 kernel: [13984019.086698] init: ganglia-monitor main process ended, respawning

The above error messages were unhelpful, so I used strace -f /usr/sbin/gmond -f, saw that gmond parses /etc/ganglia/conf.d/* before aborting, and then did a binary search to remove files in conf.d/ until the problem disappeared.

Because new images were not built, I tried to work around this myself like so:

sudo mv /etc/apt/apt.conf.d/apt.conf.d/* /etc/apt/apt.conf.d/
sudo mv /etc/apt/preferences.d/preferences.d/* /etc/apt/preferences.d/
sudo mv /etc/apt/sources.list.d/sources.list.d/* /etc/apt/sources.list.d/
sudo rmdir /etc/apt/apt.conf.d/apt.conf.d/
sudo rmdir /etc/apt/preferences.d/preferences.d/
sudo rmdir /etc/apt/sources.list.d/sources.list.d/

However that results in this error:

N: Ignoring file 'puppet_base_2.7' in directory '/etc/apt/preferences.d/' as it has an invalid filename extension

So I gave it the proper extension:

sudo mv /etc/apt/preferences.d/puppet_base_2.7{,.pref}

However that installs puppet 2.7.11-1ubuntu2 instead of 3.4.3-1~ubuntu12.04.1. Puppet 2.7's version of the 'exec' type doesn't support the 'umask' attribute, resulting in this cryptic error when I apply role::puppet::self:

err: Failed to apply catalog: Invalid parameter umask at /etc/puppet/modules/git/manifests/clone.pp:147

Solution:

sudo rm /etc/apt/preferences.d/puppet_base_2.7.pref

I'm surprised to report that this is still happening on instances created after the patch was merged. I tried twice.

This is broken again since June 8:

Strongswan 5.3.0-1+wmf2 is currently in our apt repo. I'll make a separate task for config values.

This machine crashed again. All the errors are on socket 0, so we should probably replace that DIMM.

a) no grub prompt
b) yes, I see kernel output
c) yes, I see getty on the serial port.

There's also a Debian bug report discussing this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=685504

I discussed this problem with a friend in neteng at Twitter, who says he has seen similar behavior in Juniper switches before. He recommends, and I agree: let's reboot the switch (asw-d2-eqiad).

Also let's try to ensure the NIC gets rebooted. My expectation is that racadm serveraction powercycle will do it. I've seen NICs get into weird states that persisted across warm reboots before.

I recommend checking all bios settings against 1035 for accidental changes (I can't imagine what setting would cause this, but we might as well rule it out), which will also result in rebooting the box to confirm that this behavior is reproducible.

This is mysterious. I've compared the problem host, analytics1036 (ge-2/0/5), with healthy analytics1035 (ge-2/0/4), which sits right next to it in rack D-2.

• Confirmed the problem:
  • Neon (row A), iron (row B), and analytics1029 (row C) can ping analytics1035 but not analytics1036.
  • Other analytics hosts in row D (analytics1035 in D-2, analytics1041 in D-4) can ping analytics1036.
  • None of the 12 other analytics hosts rebooted today have this problem
  • It's possible to reach the affected host from another host in the same rack. ssh -A through iron -> analytics1035 -> analytics1036 works.
• Same running kernel
• Both hosts rebooted today
• ip addr show output looks the same: same netmasks etc.
• netstat -rn looks the same
• /etc/network/interfaces looks the same
• lldpctl output looks the same
• On asw-d-eqiad, show interfaces for ge-2/0/4 and ge-2/0/5 look the same and show vlans analytics1-d-eqiad shows both ports as members
  • The only thing that looks odd to me is this part of the running config which seems to have some redundancy and doesn't treat all ports consistently (2/0/3 and 2/0/7 not in 'member-range', 4/0/* not in 'member', but those are not the ports in question!):

interface-range vlan-analytics1-d-eqiad {
    member "ge-2/0/[0-3]";
    member "ge-2/0/[4-6]";
    member ge-2/0/7;
    member-range ge-2/0/0 to ge-2/0/2;
    member-range ge-2/0/4 to ge-2/0/6;
    member-range ge-4/0/1 to ge-4/0/4;
    unit 0 {
        family ethernet-switching {
            vlan {
                members analytics1-d-eqiad;
            }
        }
    }
}

I followed this: https://git.wikimedia.org/blob/operations%2Fpuppet.git/2cdd08f9686b040816bd0dd8e63e712f4b084a7a/modules%2Fpackage_builder%2FREADME.md

Deployed, tested, documented: https://wikitech.wikimedia.org/wiki/IPsec#Emergency_shutdown

Patch: https://gerrit.wikimedia.org/r/#/c/211931/

Eventually I'd like to see the partman receipe fixed and tested by reinstalling one of these hosts, but I've fixed the running config so that the arrays no longer appear as degraded:

gage@logstash1006:~$ cat /proc/mdstat
Personalities : [raid1] [raid0]
md0 : active raid1 sda2[0] sdb2[1]
      249869312 blocks super 1.2 [4/2] [UU__]
      bitmap: 2/2 pages [8KB], 65536KB chunk

This kernel is now installed on berkelium & curium.

• IPsec ESNs work (fixed in 3.19.3)
• Aesni security patch for CVE-2015-3331 is included (fixed in 3.19.3)
• Aes256gcm does not work. (fixed in 4.0, but we don't care because we plan to use aes128gcm which works in 3.19.)

Relatedly, I have learned that the reason must be quoted or only get the first word is stored:

manifests/role/cache.pp has been refactored into modules/role/manifests/cache/* which reference hieradata/common/cache/*, hence the redundant data described in this task is eliminated.

Thanks, Brandon. I'll reply in order:

To summarize remaining work:

• Strongswan 5.3.0 is needed but is currently only in Experimental. It won't be coming to Jessie so it needs to be imported to WMF's apt repo.
• Determine appropriate values for prod: lifetime, margin, auto.

The token-based solution (Proposal 1) sounds good to me; it seems like the only barrier to adoption is making a policy decision to go with a proposal which doesn't support Precise, correct?

This seems to be fixed in linux-image-3.19.0-trunk-amd64 version 3.19.3-1~exp1, currently in Debian/Experimental.

Done!

Hi Marcel,

Added joal to eventlogging-roots + eventlogging-admins to match nuria, millimetric, mforns. This gives him sudo on eventlog1001 as well as access to hafnium.

Berkelium and Curium are now upgraded to Debian's 3.19.3 kernels containing the IPsec patch. Next, I will test the aes256gcm and ESN behavior.

Strongswan 5.3.0 has been uploaded to Debian/experimental, and is now running on Berkelium & Curium. So far the problem has not recurred.

Ok, good news. Further discussion with ecdsa has revealed that this problem is fixed in 5.3.0, which is released but not yet packaged for Debian.

I reduced some timeouts in order to recreate the problem; config changes suggested by ecdsa have not yet been made:

conn %default
        ikelifetime=6m
        lifetime=6m
        rekeyfuzz=0%
        margintime=1m
        keyingtries=%forever

Which quickly triggered the behavior. This time it's the IPv6 connection which is not established:

gage@curium:~$ sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.2.1, Linux 3.19.0-trunk-amd64, x86_64):
  uptime: 22 minutes, since Apr 15 13:13:53 2015
  malloc: sbrk 2580480, mmap 0, used 381120, free 2199360
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 2
  loaded plugins: charon aes rc2 sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default stroke updown
Listening IP addresses:
  10.64.0.170
  2620:0:861:101:862b:2bff:fefd:be6d
  2620:0:861:101:10:64:0:170
Connections:
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4:  10.64.0.170...10.64.0.169  IKEv1/2
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4:   local:  [CN=curium.eqiad.wmnet] uses public key authentication
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4:    cert:  "CN=curium.eqiad.wmnet"
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4:   remote: [CN=berkelium.eqiad.wmnet] uses public key authentication
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4:   child:  dynamic === dynamic TRANSPORT
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv6:  2620::861:101:10:64:0:170...2620::861:101:10:64:0:169  IKEv1/2
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv6:   local:  [CN=curium.eqiad.wmnet] uses public key authentication
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv6:    cert:  "CN=curium.eqiad.wmnet"
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv6:   remote: [CN=berkelium.eqiad.wmnet] uses public key authentication
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv6:   child:  dynamic === dynamic TRANSPORT
Security Associations (1 up, 0 connecting):
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4[9]: ESTABLISHED 2 minutes ago, 10.64.0.170[CN=curium.eqiad.wmnet]...10.64.0.169[CN=berkelium.eqiad.wmnet]
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4[9]: IKEv2 SPIs: 46a8b8b8776b91a6_i 18deeb67a2053fe6_r*, public key reauthentication in 2 minutes
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4[9]: IKE proposal: AES_GCM_16_128/PRF_HMAC_SHA2_384/ECP_384_BP
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4{9}:  INSTALLED, TRANSPORT, ESP SPIs: c24cb949_i c533bc7c_o
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4{9}:  AES_GCM_16_128, 0 bytes_i, 0 bytes_o, rekeying in 2 minutes
curium.eqiad.wmnet-berkelium.eqiad.wmnet_by_ipv4{9}:   10.64.0.170/32 === 10.64.0.169/32

I spoke with Tobias from Strongswan on IRC about this:

Patches merged. User account is created and access is granted.

Discussed in IRC; ran this at approximately UTC 22:32:

sudo stop carbon/relay
sudo stop carbon/cache NAME=a
sudo stop carbon/cache NAME=b
sudo stop carbon/cache NAME=c
sudo stop carbon/cache NAME=d
sudo stop carbon/cache NAME=e
sudo stop carbon/cache NAME=f
sudo stop carbon/cache NAME=g
sudo stop carbon/cache NAME=h
find /var/lib/carbon/whisper/{cassandra,restbase} -type f -name 'sample_rate.wsp' -printf "%h\n" | while read i; do sudo mv "$i/rate.wsp" "$i"/sample_rate.wsp ; done
sudo start carbon/cache NAME=a
sudo start carbon/cache NAME=b
sudo start carbon/cache NAME=c
sudo start carbon/cache NAME=d
sudo start carbon/cache NAME=e
sudo start carbon/cache NAME=f
sudo start carbon/cache NAME=g
sudo start carbon/cache NAME=h
sudo start carbon/relay

Patch is submitted: https://gerrit.wikimedia.org/r/#/c/199787/

Thanks for the feedback. Steps to reproduce are in the task description, I used IPv4:

while true ; do wget -nv -O /dev/null http://10.64.0.170/index.nginx-debian.html ; sleep 1 ; done

With the following ciphers in /etc/ipsec.conf:

ike=aes128gcm16-null-prfsha384-ecp384bp!
esp=aes128gcm16-null-ecp384bp-esn!

Ok, this is reproducible and seems to be the primary problem I was having yesterday: enabling Extended Sequence Numbers (ESN, http://kernelnewbies.org/Linux_2_6_39#head-87ffd4407af29460251c521e0228fe0ac9219d4b) causes a crash within 5 wgets.

Got another one: I changed ciphers on berkelium & restarted the daemon there; before I had a chance to restart the daemon on curium for corresponding change, curium panicked. admittedly this is not a circumstance we'll often encounter in prod:

I will switch the hadoop active namenode from analytics1001 to analytics1002 (which is in another rack) before this replacement action is taken.

I have:

1. Discussed this with @JohnLewis
2. Re-added Mjbmri@gmail.com as a list admin
3. Changed the list admin password and emailed it to Mjbmr and Atgigabyte

Not sure, as @faidon declined my ticket to monitor the Netapp stats with LibreNMS. So those stats are still only being collected by Torrus.

This order arrived at 200 Paul today.

If implemented, this task will be completed after T92604. Once we are confident in traffic flows over IPsec we may wish to use firewall rules to explicitly disallow unencrypted traffic.